[
https://issues.apache.org/jira/browse/PHOENIX-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788706#comment-17788706
]
Istvan Toth edited comment on PHOENIX-7118 at 11/22/23 1:56 PM:
----------------------------------------------------------------
revisiting the shaded beanutils and configuration2 conflicts issues:
* commons-configuration2
** hbase (2.1+): transitive only from Hadoop
** phoenix (5.1.0 + ) : 2.2.1
** omid: NOT USED
** hadoop (3.1): 2.1.1
* commons-beanutils
** hbase : transitive only from hadoop
** phoenix (5.1.0 + ) : only transitive
** omid (1.0.2): 1.9.4
** hadoop: 3.0: 1.9.3 | 3.1: 1.9.4
This does not mesh with that I wrote in PHOENIX-6053 .
configuration2 seems to use compatible versions.
Beanutils is also at least 1.9.3 ever since HBase 2.1.
It transitively gets 1.8.0 when built with Hadoop2, but that is not supported
anyway.
So it looks like the version differences are not significant enough to cause
problems.
However, the shading still can.
*Hadoop metrics2* leaks *configuration2* in the MetricsSink API, so we need to
make sure that
if we do not include Hadoop, then we shade according to the Hadoop on the
classpath.
hbase-shaded-client uses:
*org/apache/hadoop/hbase/shaded/org/apache/commons/configuration2/*
hbase-shaded-client-byo-hadoop uses: {*}not present{*}, taken from Hadoop on
classpath
hbase-shaded-mapreduce uses: {*}not present{*}, taken from Hadoop on classpath
hadoop-client-runtime uses:
*org/apache/hadoop/shaded/org/apache/commons/configuration2*
Apparently Spark 3.2+ uses hbase-client-api, while older Spark versions use
unshaded Hadoop.
Hive seems to use the standard (unshaded) Hadoop libraries even in HEAD.
For beanutils:
hbase-shaded-client uses:
*org/apache/hadoop/hbase/shaded/org/apache/commons/beanutils/*
hbase-shaded-client-byo-hadoop uses: {*}not present{*}, taken from Hadoop on
classpath
hbase-shaded-mapreduce uses: {*}not present{*}, taken from Hadoop on classpath
However, neither HBase 2, nor Hadoop3 actually use commons-beanutils, Hadoop3
just depends on it for no reason.
So it looks like commons-beanutils does not actually need any special handling,
apart from making sure that the included version is new enough for Omid.
was (Author: stoty):
revisiting the shaded beanutils and configuration2 conflicts issues:
* commons-configuration2
** hbase (2.1+): transitive only from Hadoop
** phoenix (5.1.0 + ) : 2.2.1
** omid: NOT USED
** hadoop (3.1): 2.1.1
* commons-beanutils
** hbase : transitive only from hadoop
** phoenix (5.1.0 + ) : only transitive
** omid (1.0.2): 1.9.4
** hadoop: 3.0: 1.9.3 | 3.1: 1.9.4
This does not mesh with that I wrote in PHOENIX-6053 .
configuration2 seems to use compatible versions.
Beanutils is also at least 1.9.3 ever since HBase 2.1.
It transitively gets 1.8.0 when built with Hadoop2, but that is not supported
anyway.
So it looks like the version differences are not significant enough to cause
problems.
However, the shading still can.
*Hadoop metrics2* leaks *configuration2* in the MetricsSink API, so we need to
make sure that
if we do not include Hadoop, then we shade according to the Hadoop on the
classpath.
hbase-shaded-client uses:
*org/apache/hadoop/hbase/shaded/org/apache/commons/configuration2/*
hbase-shaded-client-byo-hadoop uses: {*}not present{*}, taken from Hadoop on
classpath
hadoop-client-runtime uses:
*org/apache/hadoop/shaded/org/apache/commons/configuration2*
Apparently Spark 3.2+ uses hbase-client-api, while older Spark versions use
unshaded Hadoop.
Hive seems to use the standard (unshaded) Hadoop libraries even in HEAD.
BeanUtils shading does not differ, both the shaded and the byo packages include
*org/apache/hadoop/hbase/shaded/org/apache/commons/beanutils/*
However, neither HBase 2, nor Hadoop3 actually use commons-beanutils, Hadoop3
just depends on it for no reason.
So it looks like commons-beanutils does not actually need any special handling,
apart from making sure that the included version is new enough for Omid.
> Fix Shading Regressions in Connectors
> -------------------------------------
>
> Key: PHOENIX-7118
> URL: https://issues.apache.org/jira/browse/PHOENIX-7118
> Project: Phoenix
> Issue Type: Bug
> Components: connectors, hive-connector, spark-connector
> Affects Versions: connectors-6.0.0
> Reporter: Istvan Toth
> Assignee: Istvan Toth
> Priority: Major
>
> It seems like sometime during the refactors, we have broken the shading,
> particularly the exclusions / provided settings.
> Only the Spark3 connector shading looks correct, both the Hive and Spark2
> connector shadings seem to have regressed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
