[
https://issues.apache.org/jira/browse/FLINK-10356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660283#comment-16660283
]
ASF GitHub Bot commented on FLINK-10356:
----------------------------------------
zhijiangW commented on a change in pull request #6705: [FLINK-10356][network]
add sanity checks to SpillingAdaptiveSpanningRecordDeserializer
URL: https://github.com/apache/flink/pull/6705#discussion_r227286887
##########
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/serialization/SpanningRecordSerializationTest.java
##########
@@ -104,11 +116,216 @@ public void testHandleMixedLargeRecords() throws
Exception {
testSerializationRoundTrip(originalRecords, segmentSize);
}
+ /**
+ * Non-spanning, deserialization reads one byte too many and succeeds.
+ */
+ @Test
+ public void testHandleDeserializingTooMuchNonSpanning1() throws
Exception {
+ expectedException.expect(IOException.class);
+ expectedException.expectMessage(" -1 remaining unread byte");
+ testHandleWrongDeserialization(new
StringValueDeserializingTooMuch("Test string"), 32 * 1024);
+ }
+
+ /**
+ * Non-spanning, serialization length is 16 (including headers),
deserialization reads one byte
+ * too many and succeeds.
+ */
+ @Test
+ public void testHandleDeserializingTooMuchNonSpanning2() throws
Exception {
+ expectedException.expect(IOException.class);
+ expectedException.expectMessage(" -1 remaining unread byte");
+ testHandleWrongDeserialization(new
StringValueDeserializingTooMuch("Test string"), 17);
+ }
+
+ /**
+ * Non-spanning, serialization length is 16 (including headers),
deserialization reads one byte
+ * too many and fails.
+ */
+ @Test
+ public void testHandleDeserializingTooMuchNonSpanning3() throws
Exception {
+ expectedException.expect(IOException.class);
+ expectedException.expectMessage(" -1 remaining unread byte");
+
expectedException.expectCause(isA(IndexOutOfBoundsException.class));
+ testHandleWrongDeserialization(new
StringValueDeserializingTooMuch("Test string"), 16);
+ }
+
+ /**
+ * Spanning, serialization length is 16 (including headers),
deserialization reads one byte
+ * too many and fails.
+ */
+ @Test
+ public void testHandleDeserializingTooMuchSpanning1() throws Exception {
+ expectedException.expect(IOException.class);
+ expectedException.expectMessage(" -1 remaining unread byte");
+ expectedException.expectCause(isA(EOFException.class));
+ testHandleWrongDeserialization(new
StringValueDeserializingTooMuch("Test string"), 15);
+ }
+
+ /**
+ * Spanning, serialization length is 16 (including headers),
deserialization reads one byte
+ * too many and fails.
+ */
+ @Test
+ public void testHandleDeserializingTooMuchSpanning2() throws Exception {
Review comment:
Also this test can be merged with above
`testHandleDeserializingTooMuchSpanning1` because of same `expectedException`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add sanity checks to SpillingAdaptiveSpanningRecordDeserializer
> ---------------------------------------------------------------
>
> Key: FLINK-10356
> URL: https://issues.apache.org/jira/browse/FLINK-10356
> Project: Flink
> Issue Type: Improvement
> Components: Network
> Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.6.0, 1.6.1, 1.7.0
> Reporter: Nico Kruber
> Assignee: Nico Kruber
> Priority: Major
> Labels: pull-request-available
>
> {{SpillingAdaptiveSpanningRecordDeserializer}} doesn't have any consistency
> checks for usage calls or serializers behaving properly, e.g. to read only as
> many bytes as available/promised for that record. At least these checks
> should be added:
> # Check that buffers have not been read from yet before adding them (this is
> an invariant {{SpillingAdaptiveSpanningRecordDeserializer}} works with and
> from what I can see, it is followed now.
> # Check that after deserialization, we actually consumed {{recordLength}}
> bytes
> ** If not, in the spanning deserializer, we currently simply skip the
> remaining bytes.
> ** But in the non-spanning deserializer, we currently continue from the
> wrong offset.
> # Protect against {{setNextBuffer}} being called before draining all
> available records
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
