Hey guys, I think SPARK-1138 should be resolved before releasing Spark 0.9.1. It's affecting multiple users ability to use Spark 0.9 with various versions of Hadoop. I have one fix but not sure if it works for others.
-Evan On Mon, Mar 24, 2014 at 5:30 PM, Tathagata Das <tathagata.das1...@gmail.com> wrote: > Hello Kevin, > > A fix for SPARK-782 would definitely simplify building against Spark. > However, its possible that a fix for this issue in 0.9.1 will break > the builds (that reference spark) of existing 0.9 users, either due to > a change in the ASM version, or for being incompatible with their > current workarounds for this issue. That is not a good idea for a > maintenance release, especially when 1.0 is not too far away. > > Can you (and others) elaborate more on the current workarounds that > you have for this issue? Its best to understand all the implications > of this fix. > > Note that in branch 0.9, it is not fixed, neither in SBT nor in Maven. > > TD > > On Mon, Mar 24, 2014 at 4:38 PM, Kevin Markey <kevin.mar...@oracle.com> wrote: >> Is there any way that [SPARK-782] (Shade ASM) can be included? I see that >> it is not currently backported to 0.9. But there is no single issue that >> has caused us more grief as we integrate spark-core with other project >> dependencies. There are way too many libraries out there in addition to >> Spark 0.9 and before that are not well-behaved (ASM FAQ recommends shading), >> including some Hive and Hadoop libraries and a number of servlet libraries. >> We can't control those, but if Spark were well behaved in this regard, it >> would help. Even for a maintenance release, and even if 1.0 is only 6 weeks >> away! >> >> (For those not following 782, according to Jira comments, the SBT build >> shades it, but it is the Maven build that ends up in Maven Central.) >> >> Thanks >> Kevin Markey >> >> >> >> >> On 03/19/2014 06:07 PM, Tathagata Das wrote: >>> >>> Hello everyone, >>> >>> Since the release of Spark 0.9, we have received a number of important bug >>> fixes and we would like to make a bug-fix release of Spark 0.9.1. We are >>> going to cut a release candidate soon and we would love it if people test >>> it out. We have backported several bug fixes into the 0.9 and updated JIRA >>> >>> accordingly<https://spark-project.atlassian.net/browse/SPARK-1275?jql=project%20in%20(SPARK%2C%20BLINKDB%2C%20MLI%2C%20MLLIB%2C%20SHARK%2C%20STREAMING%2C%20GRAPH%2C%20TACHYON)%20AND%20fixVersion%20%3D%200.9.1%20AND%20status%20in%20(Resolved%2C%20Closed)>. >>> >>> Please let me know if there are fixes that were not backported but you >>> would like to see them in 0.9.1. >>> >>> Thanks! >>> >>> TD >>> >> -- -- Evan Chan Staff Engineer e...@ooyala.com |
