GitHub user gaborgsomogyi opened a pull request:
https://github.com/apache/spark/pull/19893
[SPARK-16139][TEST] Add logging functionality for leaked threads in tests
## What changes were proposed in this pull request?
Lots of our tests don't properly shutdown everything they create, and end
up leaking lots of threads. For example, `TaskSetManagerSuite` doesn't stop the
extra `TaskScheduler` and `DAGScheduler` it creates. There are a couple more
instances, eg. in `DAGSchedulerSuite`.
This PR adds the possibility to print out the not properly stopped thread
list after a test suite executed. The format is the following:
```
===== FINISHED o.a.s.scheduler.DAGSchedulerSuite: 'task end event should
have updated accumulators (SPARK-20342)' =====
...
===== Global thread whitelist loaded with name /thread_whitelist from
classpath: rpc-client.*, rpc-server.*, shuffle-client.*, shuffle-server.*' =====
ScalaTest-run:
===== THREADS NOT STOPPED PROPERLY =====
ScalaTest-run: dag-scheduler-event-loop
ScalaTest-run: globalEventExecutor-2-5
ScalaTest-run:
===== END OF THREAD DUMP =====
ScalaTest-run:
===== EITHER PUT THREAD NAME INTO THE WHITELIST FILE OR SHUT IT DOWN
PROPERLY =====
```
With the help of this leaking threads has been identified in
TaskSetManagerSuite. My intention is to hunt down and fix such bugs in later
PRs.
## How was this patch tested?
Manual: TaskSetManagerSuite test executed and found out where are the
leaking threads.
Automated: Pass the Jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gaborgsomogyi/spark SPARK-16139
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19893.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19893
----
commit 9acbb62f90656321f6db3a52e5c69fd1745dba7f
Author: Gabor Somogyi <[email protected]>
Date: 2017-12-05T08:52:55Z
[SPARK-16139][TEST] Add logging functionality for leaked threads in tests
commit 9058603a815fac49af969ff595b43223b6f34f73
Author: Gabor Somogyi <[email protected]>
Date: 2017-12-05T09:16:07Z
Apache license header added
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
