[ 
https://issues.apache.org/jira/browse/NIFI-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arvind Kandpal updated NIFI-16610:
----------------------------------
    Description: 
StreamDemarcator does not find a multi-byte delimiter when the bytes just 
before it contain a partial match of the delimiter.

In nextToken(), when a byte does not match delimiterBytes[j], the match index j 
is reset to 0. It should fall back to the longest part of the delimiter that is 
still matched (as in the KMP algorithm). Because of this, a delimiter that 
starts inside a partial match is skipped, and the two tokens are returned as 
one.

StreamDemarcator is used by PublishKafka (DelimitedStreamKafkaRecordConverter) 
when the Message Demarcator property is set. With a multi-byte demarcator, 
messages can be silently combined into one Kafka message.

Steps to reproduce (StreamDemarcator with maxDataSize 1000):
{noformat}
data "msg1\r\r\nmsg2"  delimiter "\r\n"     actual [msg1\r\r\nmsg2]  expected 
[msg1\r, msg2]
data "x---END--y"      delimiter "--END--"  actual [x---END--y]      expected 
[x-, y]
data "helloaaab"       delimiter "aab"      actual [helloaaab]       expected 
[helloa]
{noformat}
Single-byte delimiters such as "\n" are not affected.

Code: nifi-utils, StreamDemarcator.java, nextToken()

The existing tests in StreamDemarcatorTest only use delimiters where this case 
does not happen (for example "daa").

  was:
StreamDemarcator does not find a multi-byte delimiter when the bytes just 
before it contain a partial match of the delimiter.

In nextToken(), when a byte does not match delimiterBytes[j], the match index j 
is reset to 0, but the current byte is not compared again with 
delimiterBytes[0]. If that byte is the start of the real delimiter, the 
delimiter is skipped and the two tokens are returned as one.

StreamDemarcator is used by PublishKafka (DelimitedStreamKafkaRecordConverter) 
when the Message Demarcator property is set. With a multi-byte demarcator, 
messages can be silently combined into one Kafka message.

Steps to reproduce (StreamDemarcator with maxDataSize 1000):
{noformat}
data "msg1\r\r\nmsg2"  delimiter "\r\n"     actual [msg1\r\r\nmsg2]  expected 
[msg1\r, msg2]
data "x---END--y"      delimiter "--END--"  actual [x---END--y]      expected 
[x-, y]
data "helloaaab"       delimiter "aab"      actual [helloaaab]       expected 
[helloa]
{noformat}

Single-byte delimiters such as "\n" are not affected.

Code: nifi-utils, StreamDemarcator.java, nextToken()

The existing tests in StreamDemarcatorTest only use delimiters where this case 
does not happen (for example "daa").


> StreamDemarcator misses a multi-byte delimiter that follows a partial match
> ---------------------------------------------------------------------------
>
>                 Key: NIFI-16610
>                 URL: https://issues.apache.org/jira/browse/NIFI-16610
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Extensions
>            Reporter: Arvind Kandpal
>            Assignee: Anshu Singh
>            Priority: Major
>
> StreamDemarcator does not find a multi-byte delimiter when the bytes just 
> before it contain a partial match of the delimiter.
> In nextToken(), when a byte does not match delimiterBytes[j], the match index 
> j is reset to 0. It should fall back to the longest part of the delimiter 
> that is still matched (as in the KMP algorithm). Because of this, a delimiter 
> that starts inside a partial match is skipped, and the two tokens are 
> returned as one.
> StreamDemarcator is used by PublishKafka 
> (DelimitedStreamKafkaRecordConverter) when the Message Demarcator property is 
> set. With a multi-byte demarcator, messages can be silently combined into one 
> Kafka message.
> Steps to reproduce (StreamDemarcator with maxDataSize 1000):
> {noformat}
> data "msg1\r\r\nmsg2"  delimiter "\r\n"     actual [msg1\r\r\nmsg2]  expected 
> [msg1\r, msg2]
> data "x---END--y"      delimiter "--END--"  actual [x---END--y]      expected 
> [x-, y]
> data "helloaaab"       delimiter "aab"      actual [helloaaab]       expected 
> [helloa]
> {noformat}
> Single-byte delimiters such as "\n" are not affected.
> Code: nifi-utils, StreamDemarcator.java, nextToken()
> The existing tests in StreamDemarcatorTest only use delimiters where this 
> case does not happen (for example "daa").



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to