[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894685#comment-15894685
]
Jason Lowe commented on HADOOP-13866:
-------------------------------------
We've never released a non-alpha version of Hadoop that pulled in
netty-4.1.0.Beta. Hadoop 2.7 has shipped netty-all 4.0.23.Final and there
hasn't been any non-alpha releases since 2.7.
I'm still a little confused about HDFS-8377. Source changes were required
before it was reverted, but it wasn't actually reverted in total? The netty
upgrade from 4.0.23.Final to 4.1.0.Beta is still there even after the "revert."
If the netty upgrade wasn't reverted, how did that solve the source
incompatibilities?
In any case, according to the [Netty documentation on
4.1|http://netty.io/wiki/new-and-noteworthy-in-4.1.html] 4.1 is not completely
backwards compatible with 4.0. It will minimally require a recompile in some
cases.
I think we all agree shipping a 4.1.0.Beta dependency is a bad idea, so it
looks like we have the following choices:
- Revert the 2.8 changes that required netty-all to be upgraded so we can go
back to 4.0.23.Final
- Shade the netty-all 4.1.1.Final dependency
- Ship the upgraded netty-all 4.1.1.Final dependency
The risk of shading is that users could have relied on the netty 4 version
being used by Hadoop in 2.7. If we shade it then those users are broken when
they move to 2.8. The risk of shipping the exposed 4.1.1.Final dependency is
that users who fall into the category of incompatibilities between 4.0 and 4.1
will break.
The safest thing to do from a 2.7->2.8 upgrade perspective is to not change the
dependency at all which means reverting changes that required 4.1. Looks like
that involves primarily HDFS-8377 (which is reverted but not reverted?) and
HDFS-9711. I'm definitely not an expert on those changes. Are they critical
to the 2.8 release? Would it make sense to move forward with 2.8 without them?
If those changes, and therefore netty 4.1, are needed for 2.8 then I'd lean
slightly towards a shading approach. It will break 2.7 users who ended up
relying on Hadoop's netty 4.1 dependency, but it will be a cleaner upgrade for
those coming from 2.6 or earlier or anyone shipping their own netty 4.x version.
> Upgrade netty-all to 4.1.1.Final
> --------------------------------
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 3.0.0-alpha1
> Reporter: Ted Yu
> Assignee: Ted Yu
> Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch, HADOOP-13866.v9.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_000005/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
