[
https://issues.apache.org/jira/browse/FLINK-36072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junrui Lee closed FLINK-36072.
------------------------------
Resolution: Fixed
master:
2d9fb799ed6647898ca353163723d3484dabfb6c
5c5b4e08179c340a69ae4fc2ba02027ec9788af7
> Optimizing the Overhead of the Network Layer in Adaptive Execution Scenarios
> ----------------------------------------------------------------------------
>
> Key: FLINK-36072
> URL: https://issues.apache.org/jira/browse/FLINK-36072
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Reporter: Junrui Li
> Assignee: Junrui Lee
> Priority: Major
> Labels: pull-request-available
>
> In adaptive execution scenarios, hash edges may transition to broadcast
> edges; however, at that point, the upstream may have already produced data
> based on hashes.
> For example, in the adaptive broadcast join case, this results in each
> downstream task needing to connect to all upstream tasks and create a
> partition reader for each subpartition, leading to significant overhead
> (O(N²)). To optimize the overhead of the network layer, we need to enable
> downstream tasks to consume all subpartitions from a specific Task Manager
> using a single channel and a single partition reader in such situations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
