cloud-fan commented on a change in pull request #24699: [SPARK-27666][CORE] Do
not release lock while TaskContext already completed
URL: https://github.com/apache/spark/pull/24699#discussion_r294172657
##########
File path: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
##########
@@ -1176,6 +1177,31 @@ class RDDSuite extends SparkFunSuite with
SharedSparkContext {
}.collect()
}
+ test("SPARK-27666: Do not release lock while TaskContext already completed")
{
+ val rdd = sc.parallelize(Range(0, 10), 1).cache()
+ val tid = sc.longAccumulator("threadId")
+ // validate cache
+ rdd.collect()
+ rdd.mapPartitions { iter =>
+ val t = new Thread(() => {
+ while (iter.hasNext) {
+ iter.next()
+ Thread.sleep(100)
+ }
+ })
+ t.setDaemon(false)
+ t.start()
+ tid.add(t.getId)
+ Iterator(0)
+ }.collect()
+ val tmx = ManagementFactory.getThreadMXBean
+ var t = tmx.getThreadInfo(tid.value)
+ // getThreadInfo() will return null after child thread `t` died
Review comment:
This reminds me of one thing: the tests are run in local mode, so driver and
executor are in the same JVM. Seems `CountDownLatch` should work? What we
really need is: the thread in the task sends a signal to the main thread.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
