[
https://issues.apache.org/jira/browse/SPARK-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Wendell resolved SPARK-1789.
------------------------------------
Resolution: Fixed
Fix Version/s: 1.0.0
Issue resolved by pull request 723
[https://github.com/apache/spark/pull/723]
> Multiple versions of Netty dependencies cause FlumeStreamSuite failure
> ----------------------------------------------------------------------
>
> Key: SPARK-1789
> URL: https://issues.apache.org/jira/browse/SPARK-1789
> Project: Spark
> Issue Type: Bug
> Components: Build
> Affects Versions: 0.9.1
> Reporter: Sean Owen
> Labels: flume, netty, test
> Fix For: 1.0.0
>
>
> TL;DR is there is a bit of JAR hell trouble with Netty, that can be mostly
> resolved and will resolve a test failure.
> I hit the error described at
> http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-startup-time-out-td1753.html
> while running FlumeStreamingSuite, and have for a short while (is it just
> me?)
> velvia notes:
> "I have found a workaround. If you add akka 2.2.4 to your dependencies, then
> everything works, probably because akka 2.2.4 brings in newer version of
> Jetty."
> There are at least 3 versions of Netty in play in the build:
> - the new Flume 1.4.0 dependency brings in io.netty:netty:3.4.0.Final, and
> that is the immediate problem
> - the custom version of akka 2.2.3 depends on io.netty:netty:3.6.6.
> - but, Spark Core directly uses io.netty:netty-all:4.0.17.Final
> The POMs try to exclude other versions of netty, but are excluding
> org.jboss.netty:netty, when in fact older versions of io.netty:netty (not
> netty-all) are also an issue.
> The org.jboss.netty:netty excludes are largely unnecessary. I replaced many
> of them with io.netty:netty exclusions until everything agreed on
> io.netty:netty-all:4.0.17.Final.
> But this didn't work, since Akka 2.2.3 doesn't work with Netty 4.x.
> Down-grading to 3.6.6.Final across the board made some Spark code not compile.
> If the build *keeps* io.netty:netty:3.6.6.Final as well, everything seems to
> work. Part of the reason seems to be that Netty 3.x used the old
> `org.jboss.netty` packages. This is less than ideal, but is no worse than the
> current situation.
> So this PR resolves the issue and improves the JAR hell, even if it leaves
> the existing theoretical Netty 3-vs-4 conflict:
> - Remove org.jboss.netty excludes where possible, for clarity; they're not
> needed except with Hadoop artifacts
> - Add io.netty:netty excludes where needed -- except, let akka keep its
> io.netty:netty
> - Change a bit of test code that actually depended on Netty 3.x, to use 4.x
> equivalent
> - Update SBT build accordingly
> A better change would be to update Akka far enough such that it agrees on
> Netty 4.x, but I don't know if that's feasible.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
