[
https://issues.apache.org/jira/browse/FLINK-33879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiang Xin updated FLINK-33879:
------------------------------
Description:
Currently, the Hybrid Shuffle can work with Memory Tier and Disk Tier together,
however, in the following scenirio the result partition would stop working.
Suppose we have a shuffle task with 2 sub partitions. The LocalBufferPool has
15 buffers, the memory tier can use at most 15-(2*(2+1)+1) = 8 buffers. If the
memory tier used up all 8 buffers and the input channel doesn't consume them
because of some problem, the disk tier can still work with 1 reserved buffer.
However, if a redistribution happens now and the pool size is decrease to less
than 8, then the BufferAccumulator can not request buffers any more, so
was:Currently, the Hybrid Shuffle can work with Memory Tier and Disk Tier
together, if the memory tier however, in the following scenirio the result
partition would stop
> Hybrid Shuffle may hang during redistribution
> ---------------------------------------------
>
> Key: FLINK-33879
> URL: https://issues.apache.org/jira/browse/FLINK-33879
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
> Reporter: Jiang Xin
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
>
> Currently, the Hybrid Shuffle can work with Memory Tier and Disk Tier
> together, however, in the following scenirio the result partition would stop
> working.
> Suppose we have a shuffle task with 2 sub partitions. The LocalBufferPool has
> 15 buffers, the memory tier can use at most 15-(2*(2+1)+1) = 8 buffers. If
> the memory tier used up all 8 buffers and the input channel doesn't consume
> them because of some problem, the disk tier can still work with 1 reserved
> buffer. However, if a redistribution happens now and the pool size is
> decrease to less than 8, then the BufferAccumulator can not request buffers
> any more, so
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
