GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/1813
[SPARK-2848] Shade Guava in uber-jars.
For further discussion, please check the JIRA entry.
This change moves Guava classes to a different package so that they don't
conflict with the user-provided Guava (or the Hadoop-provided one). Since one
class (Optional) was exposed through Spark's public API, that class was forked
from Guava at the current dependency version (14.0.1) so that it can be kept
going forward (until the API is cleaned).
Since sbt doesn't provide the relocation feature maven-shade-plugin does, I
implemented the functionality (based on the souce for maven's plugin).
Note this change has a few implications:
- *all* classes in the final jars will reference the relocated classes. If
Hadoop classes are included (i.e. "-Phadoop-provided" is not activated), those
will also reference the Guava 14 classes (instead of the Guava 11 classes from
the Hadoop classpath).
- if the Guava version in Spark is ever changed, the new Guava will still
reference the forked Optional class; this may or may not be a problem, but in
the long term it's better to think about removing Optional from the public API.
For the end user, there are two visible implications:
- Guava is not provided as a transitive dependency anymore (since it's
"provided" in Spark)
- At runtime, unless they provide their own, they'll either have no Guava
or Hadoop's version of Guava (11), depending on how they set up their classpath.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-2848
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1813.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1813
----
commit 616998ee8431e5a9becaeaec7df2e44520abc624
Author: Marcelo Vanzin <[email protected]>
Date: 2014-08-05T20:34:22Z
Shade Guava in the maven build, fork Guava's Optional.java.
commit 2fec9903c818d14ac79d3bcc2bf9d2ef7c1ca4fc
Author: Marcelo Vanzin <[email protected]>
Date: 2014-08-05T20:34:43Z
Shade Guava in the sbt build.
commit 637189bd81ef2caf865292d2e7db7e7978d463c9
Author: Marcelo Vanzin <[email protected]>
Date: 2014-08-06T19:25:23Z
Add hacky filter to prefer Spark's copy of Optional.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
