GitHub user jukkavisma opened a pull request:
https://github.com/apache/spark/pull/14831
[SPARK-17255] [Core] Spark queries inside Futures occasionally fail due to
missing class definitions
## What changes were proposed in this pull request?
Use
- getClass.getClassLoader instead of
Thread.getCurrentThread.getContextClassLoader
- Utils.getSparkClassLoader instead of Utils.getContextOrSparkClassLoader
to avoid situations where a thread from a thread pool does not have Spark
jars in its class loader.
When Spark queries are run inside Futures using
**scala.concurrent.ExecutionContext.Implicits.global** the threads offered by
ForkJoinPool are occasionally not ones that have loaded Spark and despite class
path definitions they might be missing Spark jars in their class loaders.
In our application we return Spark queries inside Futures. The query would
sometimes fail as a result of packages missing from the thread's context class
loader. There is no single point of failure. Changing the query does not help.
If the jars are missing from the thread's class loader the first attempt to
call ClassLoader.loadClass would crash the driver.
## How was this patch tested?
Tests were run (./dev/run-tests) and the Spark driver in our application is
now stable. Previously the Spark driver would crash often due to missing jars
in class loader. All queries are returned wrapped inside futures.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jukkavisma/spark SPARK-17255
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14831.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14831
----
commit 22baddea6a33f1510b3fc2762a14004398af22ac
Author: Jukka-Pekka Väärämäki <[email protected]>
Date: 2016-08-26T11:42:50Z
SPARK-17255
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
