Yingjie Cao created FLINK-15140:
-----------------------------------
Summary: Shuffle data compression does not work with
BroadcastRecordWriter.
Key: FLINK-15140
URL: https://issues.apache.org/jira/browse/FLINK-15140
Project: Flink
Issue Type: Bug
Components: Runtime / Network
Affects Versions: 1.10.0
Reporter: Yingjie Cao
Fix For: 1.10.0
I tested the newest code of master branch last weekend with more test cases.
Unfortunately, several problems were encountered, including a bug of
compression.
When BroadcastRecordWriter is used, for pipelined mode, because the compressor
copies the data back to the input buffer, however, the underlying buffer is
shared when BroadcastRecordWriter is used. So we can not copy the compressed
buffer back to the input buffer if the underlying buffer is shared. For
blocking mode, we wrongly recycle the buffer when buffer is not compressed, and
the problem is also triggered when BroadcastRecordWriter is used.
To fix the problem, for blocking shuffle, the reference counter should be
maintained correctly, for pipelined shuffle, the simplest way maybe disable
compression is the underlying buffer is shared. I will open a PR to fix the
problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
