[
https://issues.apache.org/jira/browse/HIVE-24484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17255634#comment-17255634
]
David Mollitor commented on HIVE-24484:
---------------------------------------
This effort is stuck on my part.
There are two big issues here:
Hive integration tests fire up Druid, Kafka, HDFS, LLAP, etc. all in the same
JVM and their 3rd party dependencies are all over the place. Using a higher
version of a dependency breaks one product, but using a lower version breaks
the other. To make this work well, there probably needs to be a way to launch
each service in their own JVM class loader. In lieu of that, I've been trying
to move the ball closer to the goal post and getting dependencies closer
together.
apache/druid#10683
HIVE-24542
In HDFS 3.3.0, Hadoop team introduced ProtobufRpcEngine2 in addition to
ProtobufRpcEngine (sigh). Some of the Hive LLAP stuff is using this Hadoop
Protobuf RPC engine (ProtobufRpcEngine). There's some static logic in the
protocol engines that prohibits loading both RPC engines into the same JVM at
the same time, I'm not sure why. HDFS was migrated to ProtobufRpcEngine2. So,
again, in the integration tests, when the HDFS mini cluster is loaded, version
2 of the RPC engine is loaded into the JVM. When LLAP is later loaded, it fails
to start because version 1 cannot be registered at the same time.
> Upgrade Hadoop to 3.3.0
> -----------------------
>
> Key: HIVE-24484
> URL: https://issues.apache.org/jira/browse/HIVE-24484
> Project: Hive
> Issue Type: Improvement
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
