@evan >From the discussion in the JIRA, it seems that we still dont have a clear solution for SPARK-1138. Nor do we have a sense of whether the solution is going to small enough for a maintenance release. So I dont think we should block the release of Spark 0.9.1 for this. We can make another Spark 0.9.2 release once the correct solution has been figured out.
@kevin I understand the problem. I will try to port the solution for master inthis PR <https://github.com/apache/spark/pull/100/> into branch 0.9. Lets see if it works out. On Tue, Mar 25, 2014 at 10:19 AM, Kevin Markey <kevin.mar...@oracle.com>wrote: > TD: > > A correct shading of ASM should only affect Spark code unless someone is > relying on ASM 4.0 in unrelated project code, in which case they can add > org.ow2.asm:asm:4.x as a dependency. > > Our short term solution has been to repackage other libraries with a 3.2 > dependency or to exclude ASM when our use of a dependent library really > doesn't need it. As you probably know, the real problem arises in > ClassVisitor, which is an Interface in 3.x and before, but in 4.x it is an > abstract class that takes a version constant as its constructor. The ASM > folks of course had our best interests in mind when they did this, > attempting to deal with the Java-version dependent changes from one ASM > release to the next. Unfortunately, they didn't change the names or > locations of their classes and interfaces, which would have helped. > > In our particular case, the only library from which we couldn't exclude > ASM was org.glassfish.jersey.containers:jersey-container-servlet:jar:2.5.1. > I added a new module to our project, including some dummy source code, > because we needed the library to be self contained, made the servlet -- > minus some unrelated transitive dependencies -- the only module dependency, > then used the Maven shade plugin to relocate "org.objectweb.asm" to an > arbitrary target. We added the new shaded module as a new project > dependency, plus the unrelated transitive dependencies excluded above. > This solved the problem. At least until we added WADL to the project. Then > we needed to deal with it on its own terms. > > As you can see, we left Spark alone in all its ASM 4.0 glory. Why? Spark > is more volatile than the other libraries. Also, the way in which we > needed to deploy Spark and other resources on our (Yarn) clusters suggested > that it would be easier to shade the other libraries. I wanted to avoid > having to install a locally patched Spark library into our build, updating > the cluster and individual developers whenever there's a new patch. > Individual developers such as me who are testing the impact of patches can > handle it, but the main build goes to Maven Central via our corporate > Artifactory mirror. > > If suddenly we had a Spark 0.9.1 with a shaded ASM, it would have no > negative impact on us. Only a positive impact. > > I just wish that all users of ASM would read FAQ entry 15!!! > > Thanks > Kevin > > > > On 03/24/2014 06:30 PM, Tathagata Das wrote: > >> Hello Kevin, >> >> A fix for SPARK-782 would definitely simplify building against Spark. >> However, its possible that a fix for this issue in 0.9.1 will break >> the builds (that reference spark) of existing 0.9 users, either due to >> a change in the ASM version, or for being incompatible with their >> current workarounds for this issue. That is not a good idea for a >> maintenance release, especially when 1.0 is not too far away. >> >> Can you (and others) elaborate more on the current workarounds that >> you have for this issue? Its best to understand all the implications >> of this fix. >> >> Note that in branch 0.9, it is not fixed, neither in SBT nor in Maven. >> >> TD >> >> On Mon, Mar 24, 2014 at 4:38 PM, Kevin Markey <kevin.mar...@oracle.com> >> wrote: >> >>> Is there any way that [SPARK-782] (Shade ASM) can be included? I see >>> that >>> it is not currently backported to 0.9. But there is no single issue that >>> has caused us more grief as we integrate spark-core with other project >>> dependencies. There are way too many libraries out there in addition to >>> Spark 0.9 and before that are not well-behaved (ASM FAQ recommends >>> shading), >>> including some Hive and Hadoop libraries and a number of servlet >>> libraries. >>> We can't control those, but if Spark were well behaved in this regard, it >>> would help. Even for a maintenance release, and even if 1.0 is only 6 >>> weeks >>> away! >>> >>> (For those not following 782, according to Jira comments, the SBT build >>> shades it, but it is the Maven build that ends up in Maven Central.) >>> >>> Thanks >>> Kevin Markey >>> >>> >>> >>> >>> On 03/19/2014 06:07 PM, Tathagata Das wrote: >>> >>>> Hello everyone, >>>> >>>> Since the release of Spark 0.9, we have received a number of important >>>> bug >>>> fixes and we would like to make a bug-fix release of Spark 0.9.1. We are >>>> going to cut a release candidate soon and we would love it if people >>>> test >>>> it out. We have backported several bug fixes into the 0.9 and updated >>>> JIRA >>>> >>>> accordingly<https://spark-project.atlassian.net/browse/ >>>> SPARK-1275?jql=project%20in%20(SPARK%2C%20BLINKDB%2C% >>>> 20MLI%2C%20MLLIB%2C%20SHARK%2C%20STREAMING%2C%20GRAPH%2C% >>>> 20TACHYON)%20AND%20fixVersion%20%3D%200.9.1%20AND%20status% >>>> 20in%20(Resolved%2C%20Closed)>. >>>> >>>> Please let me know if there are fixes that were not backported but you >>>> would like to see them in 0.9.1. >>>> >>>> Thanks! >>>> >>>> TD >>>> >>>> >
