[
https://issues.apache.org/jira/browse/FLINK-33743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuxin Tan updated FLINK-33743:
------------------------------
Description: In Flink jobs that use the AdaptiveBatchScheduler and enable
adaptive parallelism, a downstream operator might consume multiple
subpartitions from an upstream operator. While downstream operators would
create an InputChannel for each upstream subpartition in Flink's current
implementation, The many InputChannels created in this situation may consume
more memory resources than needed, affecting the usability of Hybrid Shuffle
and AdaptiveBatchScheduler. In order to solve this problem, we plan to allow
one InputChannel to consume multiple subpartitions. (was: At present, a
downstream channel is limited to consuming data from a single subpartition, a
constraint that can lead to increased memory consumption. Addressing this issue
is also a critical step in ensuring that Hybrid Shuffle functions effectively
with Adaptive Query Execution (AQE). )
> Support consuming multiple subpartitions on a single channel
> ------------------------------------------------------------
>
> Key: FLINK-33743
> URL: https://issues.apache.org/jira/browse/FLINK-33743
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Reporter: Yuxin Tan
> Priority: Major
>
> In Flink jobs that use the AdaptiveBatchScheduler and enable adaptive
> parallelism, a downstream operator might consume multiple subpartitions from
> an upstream operator. While downstream operators would create an InputChannel
> for each upstream subpartition in Flink's current implementation, The many
> InputChannels created in this situation may consume more memory resources
> than needed, affecting the usability of Hybrid Shuffle and
> AdaptiveBatchScheduler. In order to solve this problem, we plan to allow one
> InputChannel to consume multiple subpartitions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
