Lijie Wang created FLINK-30631:
----------------------------------
Summary: Limit the max number of subpartitons consumed by each
downstream task
Key: FLINK-30631
URL: https://issues.apache.org/jira/browse/FLINK-30631
Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination
Reporter: Lijie Wang
Fix For: 1.17.0
In the current implementation(FLINK-25035), when the upstream vertex
parallelism is much greater than the downstream vertex parallelism, it may lead
to a large number of channels in the downstream tasks(for example, A -> B, all
to all edge, max parallelism is 1000. If parallelism of A is 1000, parallelism
of B is decided to be 1, then the only subtask of B will consume 1000 * 1000
subpartitions), resulting in a large overhead for processing channels.
In this ticket, we temporarily address this issue by limiting the max number of
subpartitons consumed by each downstream task. The ultimate solution should be
to support single channel consume multiple subpartitons.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
