Yuxin Tan created FLINK-33742:
---------------------------------
Summary: Hybrid Shuffle should work well with Adaptive Query
Execution
Key: FLINK-33742
URL: https://issues.apache.org/jira/browse/FLINK-33742
Project: Flink
Issue Type: Improvement
Components: Runtime / Coordination, Runtime / Network
Affects Versions: 1.19.0
Reporter: Yuxin Tan
At present, Hybrid Shuffle and Adaptive Query Execution (AQE), which includes
features such as Dynamic Partition Pruning (DPP), Runtime Filter, and Adaptive
Batch Scheduler, are not fully compatible. While they can be used concurrently
at the same time, the activation of AQE inhibits the key capability of Hybrid
Shuffle to perform simultaneous reading and writing. This limitation arises
because AQE dictates that downstream tasks may only initiate once upstream
tasks have finished, a requirement that is inconsistent with the simultaneous
read-write process facilitated by Hybrid Shuffle.
To harness the full potential of Hybrid Shuffle and AQE, it is essential to
refine their integration. By doing so, we can capitalize on each feature's
distinct advantages and enhance overall system performance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
