[
https://issues.apache.org/jira/browse/FLINK-29053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song closed FLINK-29053.
--------------------------------
Resolution: Fixed
master (1.16): 8b8245ba46b25c2617d91cff3d3a44b99879d9f2
> Hybrid shuffle has concurrent modification of buffer when compression is
> enabled
> --------------------------------------------------------------------------------
>
> Key: FLINK-29053
> URL: https://issues.apache.org/jira/browse/FLINK-29053
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
> Affects Versions: 1.16.0
> Reporter: Weijie Guo
> Assignee: Weijie Guo
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> When the downstream thread obtains the buffer and consuming it, if the data
> is compressed in the spilling thread and copied to the original buffer in the
> same time, since the two threads share the same memory data, the consuming
> thread will consume incorrect data, causing problems such as deserialize the
> data disorder.
> Considering that the downstream consumption is prohibited during compression,
> or block spilling thread when the downstream consumption is not completed
> will have a great impact on performance. I think we should move the
> compression operation to the write thread and store the compressed buffer
> directly in memory.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
