TD:
A correct shading of ASM should only affect Spark code unless someone is
relying on ASM 4.0 in unrelated project code, in which case they can add
org.ow2.asm:asm:4.x as a dependency.
Our short term solution has been to repackage other libraries with a 3.2
dependency or to exclude ASM when our use of a dependent library really
doesn't need it. As you probably know, the real problem arises in
ClassVisitor, which is an Interface in 3.x and before, but in 4.x it is
an abstract class that takes a version constant as its constructor. The
ASM folks of course had our best interests in mind when they did this,
attempting to deal with the Java-version dependent changes from one ASM
release to the next. Unfortunately, they didn't change the names or
locations of their classes and interfaces, which would have helped.
In our particular case, the only library from which we couldn't exclude
ASM was
org.glassfish.jersey.containers:jersey-container-servlet:jar:2.5.1. I
added a new module to our project, including some dummy source code,
because we needed the library to be self contained, made the servlet --
minus some unrelated transitive dependencies -- the only module
dependency, then used the Maven shade plugin to relocate
"org.objectweb.asm" to an arbitrary target. We added the new shaded
module as a new project dependency, plus the unrelated transitive
dependencies excluded above. This solved the problem. At least until
we added WADL to the project. Then we needed to deal with it on its own
terms.
As you can see, we left Spark alone in all its ASM 4.0 glory. Why?
Spark is more volatile than the other libraries. Also, the way in which
we needed to deploy Spark and other resources on our (Yarn) clusters
suggested that it would be easier to shade the other libraries. I
wanted to avoid having to install a locally patched Spark library into
our build, updating the cluster and individual developers whenever
there's a new patch. Individual developers such as me who are testing
the impact of patches can handle it, but the main build goes to Maven
Central via our corporate Artifactory mirror.
If suddenly we had a Spark 0.9.1 with a shaded ASM, it would have no
negative impact on us. Only a positive impact.
I just wish that all users of ASM would read FAQ entry 15!!!
Thanks
Kevin
On 03/24/2014 06:30 PM, Tathagata Das wrote:
Hello Kevin,
A fix for SPARK-782 would definitely simplify building against Spark.
However, its possible that a fix for this issue in 0.9.1 will break
the builds (that reference spark) of existing 0.9 users, either due to
a change in the ASM version, or for being incompatible with their
current workarounds for this issue. That is not a good idea for a
maintenance release, especially when 1.0 is not too far away.
Can you (and others) elaborate more on the current workarounds that
you have for this issue? Its best to understand all the implications
of this fix.
Note that in branch 0.9, it is not fixed, neither in SBT nor in Maven.
TD
On Mon, Mar 24, 2014 at 4:38 PM, Kevin Markey <kevin.mar...@oracle.com> wrote:
Is there any way that [SPARK-782] (Shade ASM) can be included? I see that
it is not currently backported to 0.9. But there is no single issue that
has caused us more grief as we integrate spark-core with other project
dependencies. There are way too many libraries out there in addition to
Spark 0.9 and before that are not well-behaved (ASM FAQ recommends shading),
including some Hive and Hadoop libraries and a number of servlet libraries.
We can't control those, but if Spark were well behaved in this regard, it
would help. Even for a maintenance release, and even if 1.0 is only 6 weeks
away!
(For those not following 782, according to Jira comments, the SBT build
shades it, but it is the Maven build that ends up in Maven Central.)
Thanks
Kevin Markey
On 03/19/2014 06:07 PM, Tathagata Das wrote:
Hello everyone,
Since the release of Spark 0.9, we have received a number of important bug
fixes and we would like to make a bug-fix release of Spark 0.9.1. We are
going to cut a release candidate soon and we would love it if people test
it out. We have backported several bug fixes into the 0.9 and updated JIRA
accordingly<https://spark-project.atlassian.net/browse/SPARK-1275?jql=project%20in%20(SPARK%2C%20BLINKDB%2C%20MLI%2C%20MLLIB%2C%20SHARK%2C%20STREAMING%2C%20GRAPH%2C%20TACHYON)%20AND%20fixVersion%20%3D%200.9.1%20AND%20status%20in%20(Resolved%2C%20Closed)>.
Please let me know if there are fixes that were not backported but you
would like to see them in 0.9.1.
Thanks!
TD
