HeartSaVioR commented on a change in pull request #25808:
[SPARK-29104][CORE][TESTS] Fix PipedRDDSuite to use `eventually` to check
thread termination
URL: https://github.com/apache/spark/pull/25808#discussion_r325022849
##########
File path: core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala
##########
@@ -100,11 +102,16 @@ class PipedRDDSuite extends SparkFunSuite with
SharedSparkContext {
assert(result.collect().length === 0)
- // collect stderr writer threads
- val stderrWriterThread = Thread.getAllStackTraces.keySet().asScala
- .find { _.getName.startsWith(PipedRDD.STDIN_WRITER_THREAD_PREFIX) }
-
- assert(stderrWriterThread.isEmpty)
+ // SPARK-29104 PipedRDD will invoke `stdinWriterThread.interrupt()` at
task completion,
+ // and `obj.wait` will get InterruptedException. However, there exists a
possibility
Review comment:
I played with debugger for the first case as second case cannot be
reproduced easily but first case would be always reproduced. It's slightly
different though no big deal.
With debugger, for normal case with current master branch,
`stdinWriterThread` catches the InterruptedException and escape run() function
after executing catch/finally statements. `stderrReaderThread` doesn't catch
the InterruptedException as it's already destroyed when we try to interrupt.
We don't join these threads for task completion anyway so it seems to be
also possible even for first case to let test code verify earlier than let
stdinWriterThread be destroyed. Maybe you've already explained same and I
misunderstood ;( If that's the case, I'm sorry to bother you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
