Eric Yang created SPARK-57041:
---------------------------------
Summary: Fix deadlock between waitForSubqueries and lazy val
initialization in subquery thread
Key: SPARK-57041
URL: https://issues.apache.org/jira/browse/SPARK-57041
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1
Reporter: Eric Yang
A deadlock can occur between a broadcast-exchange thread and a subquery thread:
- Thread A (broadcast-exchange): enters SparkPlan.waitForSubqueries(), which
acquires this.synchronized, then blocks waiting for a subquery future.
- Thread B (subquery-0): while executing the subquery under AQE, onUpdatePlan
calls SparkPlanInfo.fromSparkPlan → FileSourceScanLike.metadata (a Scala lazy
val). Scala compiles lazy val to this.synchronized, so Thread B blocks trying
to acquire the same lock Thread A holds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
