[
https://issues.apache.org/jira/browse/SPARK-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-6029:
-----------------------------
Priority: Minor (was: Major)
Summary: Unshaded "clearspring" classpath leakage + excluded fastutil
interferes with apps using clearspring (was: Spark excludes "fastutil"
dependencies of "clearspring" quantiles)
> Unshaded "clearspring" classpath leakage + excluded fastutil interferes with
> apps using clearspring
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-6029
> URL: https://issues.apache.org/jira/browse/SPARK-6029
> Project: Spark
> Issue Type: Bug
> Components: Build
> Affects Versions: 1.2.1
> Reporter: Jim Kleckner
> Priority: Minor
>
> Spark includes the clearspring analytics package but intentionally excludes
> the dependencies of the fastutil package.
> Spark includes parquet-column which includes fastutil and relocates it under
> parquet/ but creates a shaded jar file which is incomplete because it shades
> out some of the fastutil classes, notably Long2LongOpenHashMap, which is
> present in the fastutil jar file that parquet-column is referencing.
> We are using more of the clearspring classes (e.g. QDigest) and those do
> depend on missing fastutil classes like Long2LongOpenHashMap.
> Even though I add them to our assembly jar file, the class loader finds the
> spark assembly and we get runtime class loader errors when we try to use it.
> The
> [documentaion|http://spark.apache.org/docs/1.2.0/configuration.html#runtime-environment]
> and possibly related issue
> [SPARK-939|https://issues.apache.org/jira/browse/SPARK-939] suggest arguments
> that I tried with spark-submit:
> {code}
> --conf spark.driver.userClassPathFirst=true \
> --conf spark.executor.userClassPathFirst=true
> {code}
> but we still get the class not found error.
> Could this be a bug with {{userClassPathFirst=true}}? i.e. should it work?
> In any case, would it be reasonable to not exclude the "fastutil"
> dependencies?
> See email discussion
> [here|http://apache-spark-user-list.1001560.n3.nabble.com/Fwd-Spark-excludes-quot-fastutil-quot-dependencies-we-need-tt21812.html]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
