Repository: spark
Updated Branches:
  refs/heads/master e904dfaf0 -> 6a2289ecf


[SPARK-23962][SQL][TEST] Fix race in currentExecutionIds().

SQLMetricsTestUtils.currentExecutionIds() was racing with the listener
bus, which lead to some flaky tests.  We should wait till the listener bus is
empty.

I tested by adding some Thread.sleep()s in SQLAppStatusListener, which
reproduced the exceptions I saw on Jenkins.  With this change, they went
away.

Author: Imran Rashid <iras...@cloudera.com>

Closes #21041 from squito/SPARK-23962.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6a2289ec
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6a2289ec
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6a2289ec

Branch: refs/heads/master
Commit: 6a2289ecf020a99cd9b3bcea7da5e78fb4e0303a
Parents: e904dfa
Author: Imran Rashid <iras...@cloudera.com>
Authored: Thu Apr 12 15:58:04 2018 +0800
Committer: Wenchen Fan <wenc...@databricks.com>
Committed: Thu Apr 12 15:58:04 2018 +0800

----------------------------------------------------------------------
 .../org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala | 1 +
 1 file changed, 1 insertion(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/6a2289ec/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
----------------------------------------------------------------------
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
index 534d8bb..dcc540f 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
@@ -34,6 +34,7 @@ trait SQLMetricsTestUtils extends SQLTestUtils {
   import testImplicits._
 
   protected def currentExecutionIds(): Set[Long] = {
+    spark.sparkContext.listenerBus.waitUntilEmpty(10000)
     statusStore.executionsList.map(_.executionId).toSet
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

Reply via email to