[
https://issues.apache.org/jira/browse/HIVE-22415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246960#comment-17246960
]
David Mollitor edited comment on HIVE-22415 at 12/10/20, 2:08 AM:
------------------------------------------------------------------
OK, just wanted to provide an update here.
I have been working hard on getting Hadoop 3.3 working with Hive, so that JDK
11 can be supported, and it's been a challenge. [HIVE-24484]
I have worked through some of the initial pain points, but I got stuck. Hadoop
introduced a new RPC mechanism using Google Protobuf v3. Some of the LLAP
stuff was built on top of the existing Hadoop RPC code with Protobuf2. It
seems that Hadoop tried to allow for interoperability between the two RPCs,
however, loading one version of the RPC engine blocks the loading of the other
one (first one wins). I think this becomes an issue for QTests since the tests
may spin up an LLAP and a Hadoop mini cluster in the same classloader context.
Simply loading the Protobuf3 Hadoop RPC (NameNode) code blocks the loading of
the Protobuf2 Hadoop RPC (LLAP) code. Without any changes on the Hadoop side
to better support this setup, the LLAP code needs to be migrated to use
Protobuf3 and to use the Hadoop 3rd part JAR with its shaded Protobuf version.
https://github.com/apache/hadoop/blob/0a45bd034e1ce08c556227bb2c815c15be17cf10/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java#L63-L67
https://github.com/apache/hadoop/blob/0a45bd034e1ce08c556227bb2c815c15be17cf10/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java#L70-L74
was (Author: belugabehr):
OK, just wanted to provide an update here.
I have been working hard on getting Hadoop 3.3 working with Hive, so that JDK
11 can be supported, and it's been a challenge. [HIVE-24484]
I have worked through some of the initial pain points, but I got stuck. Hadoop
introduced a new RPC mechanism using Google Protobuf v3. Some of the LLAP
stuff was built on top of the existing Hadoop RPC code with Protobuf2. It
seems that Hadoop tried to all for interoperability between the two RPCs,
however, loading one version of the RPC engine blocks the loading of the other
one (first one wins). I think this becomes an issue for QTests since the tests
may spin up an LLAP and a Hadoop mini cluster in the same classloader context.
Simply loading the Protobuf3 Hadoop RPC (NameNode) code blocks the loading of
the Protobuf2 Hadoop RPC (LLAP) code. Without any changes on the Hadoop side
to better support this setup, the LLAP code needs to be migrated to use
Protobuf3 and to use the Hadoop 3rd part JAR with its shaded Protobuf version.
https://github.com/apache/hadoop/blob/0a45bd034e1ce08c556227bb2c815c15be17cf10/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java#L63-L67
https://github.com/apache/hadoop/blob/0a45bd034e1ce08c556227bb2c815c15be17cf10/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java#L70-L74
> Upgrade to Java 11
> ------------------
>
> Key: HIVE-22415
> URL: https://issues.apache.org/jira/browse/HIVE-22415
> Project: Hive
> Issue Type: Improvement
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Critical
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Upgrade Hive to Java JDK 11
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
