Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13263 )
Change subject: KUDU-2775: Deflake DefaultSourceTest repartition tests ...................................................................... Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/13263/2/java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/DistributedDataGeneratorTest.scala File java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/DistributedDataGeneratorTest.scala: http://gerrit.cloudera.org:8080/#/c/13263/2/java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/DistributedDataGeneratorTest.scala@125 PS2, Line 125: def testNumTasksRepartition(): Unit = { > Isn't this test vulnerable too? True, I think the extra call to open the table and create the KuduPartitioner is slow enough it won't happen though. I will add the guard regardless. http://gerrit.cloudera.org:8080/#/c/13263/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala File java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala: http://gerrit.cloudera.org:8080/#/c/13263/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala@472 PS1, Line 472: new KuduWriteOptions(repartition = true, repartitionSort = repartitionSort)) > So if I'm understanding you correctly, when insertRows() returns we're guar Right, this is a race between a job ending and the Spark listener call completing. insertRows just starts a Spark job. Looking a bit deeper it appears that Spark implements the SparkListener via a SparkListenerBus. The SparkListenerBus uses an AsyncEventQueue to call the various `onEvent` methods. http://gerrit.cloudera.org:8080/#/c/13263/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala@476 PS1, Line 476: rrid > Isn't jobDone only set when the job ends, and the job will only end after t The timer doesn't start until after insertRows returns. So the race/timer is just for the SparkListener. -- To view, visit http://gerrit.cloudera.org:8080/13263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2302170df3bf3ebac6cc06381d764419c2d48303 Gerrit-Change-Number: 13263 Gerrit-PatchSet: 2 Gerrit-Owner: Grant Henke <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Grant Henke <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Tue, 07 May 2019 21:10:53 +0000 Gerrit-HasComments: Yes
