XiDuo You created SPARK-36085:
---------------------------------
Summary: Make broadcast query stage executionContext isolation
from AQE
Key: SPARK-36085
URL: https://issues.apache.org/jira/browse/SPARK-36085
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 3.1.2, 3.2.0, 3.3.0
Reporter: XiDuo You
`AdaptiveSparkPlanExec` shares the execution context with
`BroadcastQueryStage`, but it has a small number thread (hard code 16).
Unfortunately, it's not enough in some complex queries and no thread is
available to cancel the broadcast stage.
In normal code path (without AQE), broadcast has it's own execution context
with a config `spark.sql.broadcastExchange.maxThreadThreshold`. It should make
sense to make broadcast query stage executionContext isolation from AQE and
respect the existed thread threshold config.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
