[
https://issues.apache.org/jira/browse/SPARK-57041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated SPARK-57041:
------------------------------
Description:
A deadlock can occur between a broadcast-exchange thread and a subquery thread:
- Thread A (broadcast-exchange): enters SparkPlan.waitForSubqueries(), which
acquires this.synchronized, then blocks waiting for a subquery future.
- Thread B (subquery-0): while executing the subquery under AQE, onUpdatePlan
calls SparkPlanInfo.fromSparkPlan → FileSourceScanLike.metadata (a Scala lazy
val). Scala compiles lazy val to this.synchronized, so Thread B blocks trying
to acquire the same lock Thread A holds.
subquery thread:
{code:java}
"subquery-0" prio=5 tid=486 BLOCKED deamon
native=false, suspended=false, block=12, wait=0
lock=org.apache.spark.sql.execution.FileSourceScanExec@6a25b17b owned by
broadcast-exchange-12-89594345-aeb3-4343-aed7-1e6399b9f2e9 (489), cpu=114,
user=80
app//org.apache.spark.sql.execution.FileSourceScanExec.metadata$lzycompute(DataSourceScanExec.scala:729)
app//org.apache.spark.sql.execution.FileSourceScanExec.metadata(DataSourceScanExec.scala:729)
app//org.apache.spark.sql.execution.SparkPlanInfo$.fromSparkPlan(SparkPlanInfo.scala:93)
{code}
broadcast-exchange thread:
{code:java}
"broadcast-exchange-12-89594345-aeb3-4343-aed7-1e6399b9f2e9" prio=5 tid=489
WAITING deamon
native=false, suspended=false, block=0, wait=1
lock=java.util.concurrent.FutureTask@282caa50 owned by null (-1), cpu=14,
user=10
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
[email protected]/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:447)
[email protected]/java.util.concurrent.FutureTask.get(FutureTask.java:190)
app//org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:331)
app//org.apache.spark.sql.execution.SubqueryExec.executeCollect(basicPhysicalOperators.scala:925)
app//org.apache.spark.sql.execution.InSubqueryExec.updateResult(subquery.scala:131)
{code}
was:
A deadlock can occur between a broadcast-exchange thread and a subquery thread:
- Thread A (broadcast-exchange): enters SparkPlan.waitForSubqueries(), which
acquires this.synchronized, then blocks waiting for a subquery future.
- Thread B (subquery-0): while executing the subquery under AQE, onUpdatePlan
calls SparkPlanInfo.fromSparkPlan → FileSourceScanLike.metadata (a Scala lazy
val). Scala compiles lazy val to this.synchronized, so Thread B blocks trying
to acquire the same lock Thread A holds.
> Fix deadlock between waitForSubqueries and lazy val initialization in
> subquery thread
> -------------------------------------------------------------------------------------
>
> Key: SPARK-57041
> URL: https://issues.apache.org/jira/browse/SPARK-57041
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.1
> Reporter: Eric Yang
> Priority: Major
>
> A deadlock can occur between a broadcast-exchange thread and a subquery
> thread:
> - Thread A (broadcast-exchange): enters SparkPlan.waitForSubqueries(), which
> acquires this.synchronized, then blocks waiting for a subquery future.
> - Thread B (subquery-0): while executing the subquery under AQE,
> onUpdatePlan calls SparkPlanInfo.fromSparkPlan → FileSourceScanLike.metadata
> (a Scala lazy val). Scala compiles lazy val to this.synchronized, so Thread B
> blocks trying to acquire the same lock Thread A holds.
> subquery thread:
> {code:java}
> "subquery-0" prio=5 tid=486 BLOCKED deamon
> native=false, suspended=false, block=12, wait=0
> lock=org.apache.spark.sql.execution.FileSourceScanExec@6a25b17b owned by
> broadcast-exchange-12-89594345-aeb3-4343-aed7-1e6399b9f2e9 (489), cpu=114,
> user=80
> app//org.apache.spark.sql.execution.FileSourceScanExec.metadata$lzycompute(DataSourceScanExec.scala:729)
> app//org.apache.spark.sql.execution.FileSourceScanExec.metadata(DataSourceScanExec.scala:729)
> app//org.apache.spark.sql.execution.SparkPlanInfo$.fromSparkPlan(SparkPlanInfo.scala:93)
> {code}
> broadcast-exchange thread:
> {code:java}
> "broadcast-exchange-12-89594345-aeb3-4343-aed7-1e6399b9f2e9" prio=5 tid=489
> WAITING deamon
> native=false, suspended=false, block=0, wait=1
> lock=java.util.concurrent.FutureTask@282caa50 owned by null (-1), cpu=14,
> user=10
> [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
> [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
> [email protected]/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:447)
> [email protected]/java.util.concurrent.FutureTask.get(FutureTask.java:190)
> app//org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:331)
> app//org.apache.spark.sql.execution.SubqueryExec.executeCollect(basicPhysicalOperators.scala:925)
> app//org.apache.spark.sql.execution.InSubqueryExec.updateResult(subquery.scala:131)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
