[ https://issues.apache.org/jira/browse/NIFI-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15555864#comment-15555864 ]
ASF subversion and git services commented on NIFI-2874: ------------------------------------------------------- Commit 5c0c1d3bb41fa75622a3a7502ccf86ec49d2fe4d in nifi's branch refs/heads/0.x from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=5c0c1d3 ] NIFI-2874: Ensure that when reading more data from an InputStream the StreamDemarcator appropriately updates the max index that can be read from the buffer Signed-off-by: Mike Moser <mose...@apache.org> This closes #1114. > StreamDemarcator can return wrong data for token > ------------------------------------------------ > > Key: NIFI-2874 > URL: https://issues.apache.org/jira/browse/NIFI-2874 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions > Reporter: Mark Payne > Assignee: Mark Payne > Priority: Critical > Fix For: 1.1.0, 0.7.1 > > > There is a case where StreamDemarcator can return the wrong data for a token. > If a token ends at the end of the buffer, and the next token is smaller than > the previous, it can result in the next token keeping part of the buffer's > content. The code below is a unit test that exposes this: > {code} > @Test > public void testOnBufferSplitNoTrailingDelimiter() throws IOException { > final byte[] inputData = "Yes\nNo".getBytes(StandardCharsets.UTF_8); > ByteArrayInputStream is = new ByteArrayInputStream(inputData); > StreamDemarcator scanner = new StreamDemarcator(is, "\n".getBytes(), > 1000, 3); > final byte[] first = scanner.nextToken(); > final byte[] second = scanner.nextToken(); > assertNotNull(first); > assertNotNull(second); > assertArrayEquals(first, new byte[] {'Y', 'e', 's'}); > assertArrayEquals(second, new byte[] {'N', 'o'}); > } > {code} > In this case, the second token, which should be 'No' comes back as 'Nos' > because it contains the 's' from the previous token. -- This message was sent by Atlassian JIRA (v6.3.4#6332)