zhijiang created FLINK-10995:
--------------------------------
Summary: Copy intermediate serialization results only once for
broadcast mode
Key: FLINK-10995
URL: https://issues.apache.org/jira/browse/FLINK-10995
Project: Flink
Issue Type: Sub-task
Components: Network
Affects Versions: 1.8.0
Reporter: zhijiang
Assignee: zhijiang
The emitted records from operator would be firstly serialized into intermediate
bytes array in {{RecordSerializer}}, then copy the intermediate results into
target buffers for different sub partitions. For broadcast mode, the same
intermediate results would be copied as many times as the number of sub
partitions, and this would affect the performance seriously in large scale jobs.
We can copy to only one target buffer which would be shared by all the sub
partitions to reduce the overheads. For emitting latency marker in broadcast
mode, we should flush the previous shared target buffers first, and then
request a new buffer for the target sub partition to send latency marker.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
