Sean Owen created SPARK-1789:
--------------------------------
Summary: Multiple versions of Netty dependencies cause
FlumeStreamSuite failure
Key: SPARK-1789
URL: https://issues.apache.org/jira/browse/SPARK-1789
Project: Spark
Issue Type: Bug
Components: Build
Affects Versions: 0.9.1
Reporter: Sean Owen
TL;DR is there is a bit of JAR hell trouble with Netty, that can be mostly
resolved and will resolve a test failure.
I hit the error described at
http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-startup-time-out-td1753.html
while running FlumeStreamingSuite, and have for a short while (is it just me?)
velvia notes:
"I have found a workaround. If you add akka 2.2.4 to your dependencies, then
everything works, probably because akka 2.2.4 brings in newer version of
Jetty."
There are at least 3 versions of Netty in play in the build:
- the new Flume 1.4.0 dependency brings in io.netty:netty:3.4.0.Final, and that
is the immediate problem
- the custom version of akka 2.2.3 depends on io.netty:netty:3.6.6.
- but, Spark Core directly uses io.netty:netty-all:4.0.17.Final
The POMs try to exclude other versions of netty, but are excluding
org.jboss.netty:netty, when in fact older versions of io.netty:netty (not
netty-all) are also an issue.
The org.jboss.netty:netty excludes are largely unnecessary. I replaced many of
them with io.netty:netty exclusions until everything agreed on
io.netty:netty-all:4.0.17.Final.
But this didn't work, since Akka 2.2.3 doesn't work with Netty 4.x.
Down-grading to 3.6.6.Final across the board made some Spark code not compile.
If the build *keeps* io.netty:netty:3.6.6.Final as well, everything seems to
work. Part of the reason seems to be that Netty 3.x used the old
`org.jboss.netty` packages. This is less than ideal, but is no worse than the
current situation.
So this PR resolves the issue and improves the JAR hell, even if it leaves the
existing theoretical Netty 3-vs-4 conflict:
- Remove org.jboss.netty excludes where possible, for clarity; they're not
needed except with Hadoop artifacts
- Add io.netty:netty excludes where needed -- except, let akka keep its
io.netty:netty
- Change a bit of test code that actually depended on Netty 3.x, to use 4.x
equivalent
- Update SBT build accordingly
A better change would be to update Akka far enough such that it agrees on Netty
4.x, but I don't know if that's feasible.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
