Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1214#discussion_r102774788
--- Diff:
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
---
@@ -98,22 +136,26 @@ void fill() throws IOException {
}
int bytesRead;
+ /*
+ * The do/while pattern is used here similar to the way it is used
in
+ * BufferedReader essentially protecting from assuming the EOS
until it
+ * actually is since not every implementation of InputStream
guarantees
+ * that bytes are always available while the stream is open.
+ */
do {
bytesRead = this.is.read(this.buffer, this.index,
this.buffer.length - this.index);
} while (bytesRead == 0);
- this.bufferLength = bytesRead != -1 ? this.index + bytesRead : -1;
- if (this.bufferLength > this.maxDataSize) {
+ this.availableBytesLength = bytesRead != -1 ? this.index +
bytesRead : -1;
+ if (this.availableBytesLength > this.maxDataSize) {
--- End diff --
This logic seems wrong to me. It is saying 'If we have more data available
in the buffer than the largest token that is allowed, we should fail?? The
following unit test fails, though I believe it should pass:
```
@Test
public void testLargeBufferSmallMaxSize() throws IOException {
final byte[] inputData = "A Great Benefit To Us
All".getBytes(StandardCharsets.UTF_8);
try (final InputStream is = new ByteArrayInputStream(inputData);
final StreamDemarcator demarcator = new StreamDemarcator(is,
"B".getBytes(StandardCharsets.UTF_8), 24, 4096)) {
final byte[] first = demarcator.nextToken();
assertNotNull(first);
assertEquals("A Great ", new String(first));
final byte[] second = demarcator.nextToken();
assertNotNull(second);
assertEquals("enefit To Us All", new String(second));
assertNull(demarcator.nextToken());
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
