GitHub user gaborgsomogyi opened a pull request: https://github.com/apache/spark/pull/19893
[SPARK-16139][TEST] Add logging functionality for leaked threads in tests ## What changes were proposed in this pull request? Lots of our tests don't properly shutdown everything they create, and end up leaking lots of threads. For example, `TaskSetManagerSuite` doesn't stop the extra `TaskScheduler` and `DAGScheduler` it creates. There are a couple more instances, eg. in `DAGSchedulerSuite`. This PR adds the possibility to print out the not properly stopped thread list after a test suite executed. The format is the following: ``` ===== FINISHED o.a.s.scheduler.DAGSchedulerSuite: 'task end event should have updated accumulators (SPARK-20342)' ===== ... ===== Global thread whitelist loaded with name /thread_whitelist from classpath: rpc-client.*, rpc-server.*, shuffle-client.*, shuffle-server.*' ===== ScalaTest-run: ===== THREADS NOT STOPPED PROPERLY ===== ScalaTest-run: dag-scheduler-event-loop ScalaTest-run: globalEventExecutor-2-5 ScalaTest-run: ===== END OF THREAD DUMP ===== ScalaTest-run: ===== EITHER PUT THREAD NAME INTO THE WHITELIST FILE OR SHUT IT DOWN PROPERLY ===== ``` With the help of this leaking threads has been identified in TaskSetManagerSuite. My intention is to hunt down and fix such bugs in later PRs. ## How was this patch tested? Manual: TaskSetManagerSuite test executed and found out where are the leaking threads. Automated: Pass the Jenkins. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gaborgsomogyi/spark SPARK-16139 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19893.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19893 ---- commit 9acbb62f90656321f6db3a52e5c69fd1745dba7f Author: Gabor Somogyi <gabor.g.somo...@gmail.com> Date: 2017-12-05T08:52:55Z [SPARK-16139][TEST] Add logging functionality for leaked threads in tests commit 9058603a815fac49af969ff595b43223b6f34f73 Author: Gabor Somogyi <gabor.g.somo...@gmail.com> Date: 2017-12-05T09:16:07Z Apache license header added ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org