Arvind Kandpal created NIFI-16610:
-------------------------------------

             Summary: StreamDemarcator misses a multi-byte delimiter that 
follows a partial match
                 Key: NIFI-16610
                 URL: https://issues.apache.org/jira/browse/NIFI-16610
             Project: Apache NiFi
          Issue Type: Bug
          Components: Extensions
            Reporter: Arvind Kandpal
            Assignee: Anshu Singh


StreamDemarcator does not find a multi-byte delimiter when the bytes just 
before it contain a partial match of the delimiter.

In nextToken(), when a byte does not match delimiterBytes[j], the match index j 
is reset to 0, but the current byte is not compared again with 
delimiterBytes[0]. If that byte is the start of the real delimiter, the 
delimiter is skipped and the two tokens are returned as one.

StreamDemarcator is used by PublishKafka (DelimitedStreamKafkaRecordConverter) 
when the Message Demarcator property is set. With a multi-byte demarcator, 
messages can be silently combined into one Kafka message.

Steps to reproduce (StreamDemarcator with maxDataSize 1000):
{noformat}
data "msg1\r\r\nmsg2"  delimiter "\r\n"     actual [msg1\r\r\nmsg2]  expected 
[msg1\r, msg2]
data "x---END--y"      delimiter "--END--"  actual [x---END--y]      expected 
[x-, y]
data "helloaaab"       delimiter "aab"      actual [helloaaab]       expected 
[helloa]
{noformat}

Single-byte delimiters such as "\n" are not affected.

Code: nifi-utils, StreamDemarcator.java, nextToken()

The existing tests in StreamDemarcatorTest only use delimiters where this case 
does not happen (for example "daa").



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to