[
https://issues.apache.org/jira/browse/FLINK-10356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber updated FLINK-10356:
--------------------------------
Description:
{{SpillingAdaptiveSpanningRecordDeserializer}} doesn't have any consistency
checks for usage calls or serializers behaving properly, e.g. to read only as
many bytes as available/promised for that record. At least these checks should
be added:
# Check that buffers have not been read from yet before adding them (this is
an invariant {{SpillingAdaptiveSpanningRecordDeserializer}} works with and from
what I can see, it is followed now.
# Check that after deserialization, we actually consumed {{recordLength}} bytes
** If not, in the spanning deserializer, we currently simply skip the
remaining bytes.
** But in the non-spanning deserializer, we currently continue from the wrong
offset.
# Protect against {{setNextBuffer}} being called before draining all available
records
was:
{{SpillingAdaptiveSpanningRecordDeserializer}} doesn't have any consistency
checks for usage calls or serializers behaving properly, e.g. to read only as
many bytes as available/promised for that record. At least these checks should
be added:
# Check that buffers have not been read from yet before adding them (this is
an invariant {{SpillingAdaptiveSpanningRecordDeserializer}} works with and
from what I can see, it is followed now}}.
# Check that after deserialization, we actually consumed {{recordLength}} bytes
** If not, in the spanning deserializer, we currently simply skip the
remaining bytes.
** But in the non-spanning deserializer, we currently continue from the wrong
offset.
# Protect against {{setNextBuffer}} being called before draining all available
records
> Add sanity checks to SpillingAdaptiveSpanningRecordDeserializer
> ---------------------------------------------------------------
>
> Key: FLINK-10356
> URL: https://issues.apache.org/jira/browse/FLINK-10356
> Project: Flink
> Issue Type: Improvement
> Components: Network
> Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.6.1, 1.7.0, 1.5.4
> Reporter: Nico Kruber
> Assignee: Nico Kruber
> Priority: Major
>
> {{SpillingAdaptiveSpanningRecordDeserializer}} doesn't have any consistency
> checks for usage calls or serializers behaving properly, e.g. to read only as
> many bytes as available/promised for that record. At least these checks
> should be added:
> # Check that buffers have not been read from yet before adding them (this is
> an invariant {{SpillingAdaptiveSpanningRecordDeserializer}} works with and
> from what I can see, it is followed now.
> # Check that after deserialization, we actually consumed {{recordLength}}
> bytes
> ** If not, in the spanning deserializer, we currently simply skip the
> remaining bytes.
> ** But in the non-spanning deserializer, we currently continue from the
> wrong offset.
> # Protect against {{setNextBuffer}} being called before draining all
> available records
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
