Stephan Ewen created FLINK-19023:
------------------------------------
Summary: Remove pruning of Record Serializer Buffer
Key: FLINK-19023
URL: https://issues.apache.org/jira/browse/FLINK-19023
Project: Flink
Issue Type: Sub-task
Components: Runtime / Network
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 1.12.0
Currently, the {{SpanningRecordSerializer}} prunes its internal serialization
buffer under special circumstances:
- The buffer becomes larger than a certain threshold (5MB)
- The full record end lines up exactly with a full buffer length (this change
got introduced at some point, it is not clear what the purpose is)
This optimization virtually never kicks in (because of the second condition)
and also seems unnecessary. There is only a single serializer on the sender
side, so this will not help to reduce the maximum memory footprint needed in
any way.
NOTE: A similar optimization on the reader side
({{SpillingAdaptiveSpanningRecordDeserializer}}) makes sense, because multiple
parallel deserializers run in order to piece together the records when
retrieving buffers from the network in arbitrary order. Truncating buffers (or
spilling) there helps reduce the maximum required memory footprint.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
