Leanken.Lin created SPARK-32615:
-----------------------------------
Summary: AQE aggregateMetrics java.util.NoSuchElementException
Key: SPARK-32615
URL: https://issues.apache.org/jira/browse/SPARK-32615
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.0
Reporter: Leanken.Lin
Reproduce Step
{code:java}
//代码占位符
sql/test-only org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite --
-z "SPARK-32573: Eliminate NAAJ when BuildSide is
EmptyHashedRelationWithAllNullKeys"
{code}
{code:java}
//代码占位符
14:40:44.089 ERROR org.apache.spark.util.Utils: Uncaught exception in thread
element-tracking-store-worker14:40:44.089 ERROR org.apache.spark.util.Utils:
Uncaught exception in thread
element-tracking-store-workerjava.util.NoSuchElementException: key not found:
12 at scala.collection.immutable.Map$Map1.apply(Map.scala:114) at
org.apache.spark.sql.execution.ui.SQLAppStatusListener.$anonfun$aggregateMetrics$11(SQLAppStatusListener.scala:257)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) at
scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) at
scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) at
scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) at
scala.collection.mutable.HashMap.foreach(HashMap.scala:149) at
scala.collection.TraversableLike.map(TraversableLike.scala:238) at
scala.collection.TraversableLike.map$(TraversableLike.scala:231) at
scala.collection.AbstractTraversable.map(Traversable.scala:108) at
org.apache.spark.sql.execution.ui.SQLAppStatusListener.aggregateMetrics(SQLAppStatusListener.scala:256)
at
org.apache.spark.sql.execution.ui.SQLAppStatusListener.$anonfun$onExecutionEnd$2(SQLAppStatusListener.scala:365)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at
org.apache.spark.util.Utils$.tryLog(Utils.scala:1971) at
org.apache.spark.status.ElementTrackingStore$$anon$1.run(ElementTrackingStore.scala:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)[info] - SPARK-32573: Eliminate NAAJ
when BuildSide is EmptyHashedRelationWithAllNullKeys (2 seconds, 14
milliseconds)
{code}
This issue is mainly because during AQE, while sub-plan changed, the metrics
update is overwrite. for example, in this UT, change from BroadcastHashJoinExec
into a LocalTableScanExec, and in the onExecutionEnd action it will try
aggregate all metrics during the execution, which will cause
NoSuchElementException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
