GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/20221
[SPARK-23019][Core] Wait until SparkContext.stop() finished in
SparkLauncherSuite
## What changes were proposed in this pull request?
In current code ,the function `waitFor` in `BaseSuite`
https://github.com/apache/spark/blob/cfcd746689c2b84824745fa6d327ffb584c7a17d/launcher/src/test/java/org/apache/spark/launcher/BaseSuite.java#L53
only wait until DAGScheduler is stopped, while SparkContext.clearActiveContext
may not be called yet.
Thus, in the Jenkins test
https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.3-test-maven-hadoop-2.6/
, `JdbcRDDSuite` failed because the previous test `SparkLauncherSuite` exit
before SparkContext.stop() is finished.
To repo:
```
$ build/sbt
> project core
> testOnly *SparkLauncherSuite *JavaJdbcRDDSuite
```
To Fix:
Wait for a reasonable amount of time to avoid creating two active
SparkContext in JVM in SparkLauncherSuite.
Can' come up with any better solution for now.
## How was this patch tested?
Unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gengliangwang/spark SPARK-23019
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20221.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20221
----
commit 16b985c1018c0f5f54ef1062bdd7960cc4a87b39
Author: Wang Gengliang <ltnwgl@...>
Date: 2018-01-10T13:12:10Z
SPARK-23019: Wait until SparkContext.stop() finished in SparkLauncherSuite
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
