Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/13263 )
Change subject: KUDU-2775: Deflake DefaultSourceTest repartition tests ...................................................................... Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/13263/2/java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/DistributedDataGeneratorTest.scala File java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/DistributedDataGeneratorTest.scala: http://gerrit.cloudera.org:8080/#/c/13263/2/java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/DistributedDataGeneratorTest.scala@125 PS2, Line 125: def testNumTasksRepartition(): Unit = { Isn't this test vulnerable too? http://gerrit.cloudera.org:8080/#/c/13263/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala File java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala: http://gerrit.cloudera.org:8080/#/c/13263/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala@472 PS1, Line 472: new KuduWriteOptions(repartition = true, repartitionSort = repartitionSort)) > It appears the SparkListener.onTaskEnd isn't synchronous and `assertEquals( So if I'm understanding you correctly, when insertRows() returns we're guaranteed that all Kudu client work is done, but behind the scenes Spark may still be in the middle of ending tasks and/or the overall job? Meaning, we can't use insertRows() as a proxy for "the job and all of its tasks are done"? That's certainly better; I was worried that insertRows() wasn't synchronous w.r.t. the Kudu work it performed. http://gerrit.cloudera.org:8080/#/c/13263/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala@476 PS1, Line 476: rrid > This should be drastically more than enough and the environment shouldn't m Isn't jobDone only set when the job ends, and the job will only end after the inserts are done? That's server-side, so couldn't TSAN affect that? -- To view, visit http://gerrit.cloudera.org:8080/13263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2302170df3bf3ebac6cc06381d764419c2d48303 Gerrit-Change-Number: 13263 Gerrit-PatchSet: 2 Gerrit-Owner: Grant Henke <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Grant Henke <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Tue, 07 May 2019 20:10:58 +0000 Gerrit-HasComments: Yes
