[
https://issues.apache.org/jira/browse/FLINK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16995511#comment-16995511
]
Yingjie Cao commented on FLINK-15140:
-------------------------------------
Fix via 5139282e7f48ece00528c6ade9480f10fa3dd54b on master.
Fix via 15a76b8fe2e5cee44af504174bcbc86dc147058a on release-1.10.
> Shuffle data compression does not work with BroadcastRecordWriter.
> ------------------------------------------------------------------
>
> Key: FLINK-15140
> URL: https://issues.apache.org/jira/browse/FLINK-15140
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
> Affects Versions: 1.10.0
> Reporter: Yingjie Cao
> Assignee: Yingjie Cao
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I tested the newest code of master branch last weekend with more test cases.
> Unfortunately, several problems were encountered, including a bug of
> compression.
> When BroadcastRecordWriter is used, for pipelined mode, because the
> compressor copies the data back to the input buffer, however, the underlying
> buffer is shared when BroadcastRecordWriter is used. So we can not copy the
> compressed buffer back to the input buffer if the underlying buffer is
> shared. For blocking mode, we wrongly recycle the buffer when buffer is not
> compressed, and the problem is also triggered when BroadcastRecordWriter is
> used.
> To fix the problem, for blocking shuffle, the reference counter should be
> maintained correctly, for pipelined shuffle, the simplest way maybe disable
> compression when the underlying buffer is shared. I will open a PR to fix the
> problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
