yihua commented on issue #6398:
URL: https://github.com/apache/hudi/issues/6398#issuecomment-1248300583
@rbtrtr To provide more info, "hudi-default.xml" is packaged from the HBase
library (shaded and packaged in the Hudi bundle) and it is required to load the
default HBase config values. To avoid the config conflicts with other HBase
libs in the classpath, Hudi repo introduces "hbase-site.xml" to override the
values ("hbase-site.xml" is loaded after "hudi-default.xml").
Check this for more details. https://github.com/apache/hudi/pull/5004 .
> Addresses HBase 2.x conflict with HBase 1.x in Hive 2: In Hudi, we newly
include hudi-common/src/main/resources/hbase-site.xml containing HBase default
configs from the hbase-common 2.4.9 we use, to override the default configs
loaded from hbase-default.xml from an older HBase version on the classpath and
to ensure correct default configs for Hudi HBase usage. In Hive, the Hive
server loads all lib jars including HBase jars with its corresponding
hbase-default.xml into class path (e.g., HBase 1.1.1), and that can cause
conflict with the hbase-default.xml in Hudi bundles (HBase 2.4.9). The
exception is thrown as follows: Caused by: java.lang.RuntimeException:
hbase-default.xml file seems to be for an older version of HBase (1.1.1), this
version is 2.4.9. Relevant logic causing such exception can be found in
HBaseConfiguration::addHbaseResources(). To get around this issue, since HBase
loads hbase-site.xml after hbase-default.xml, we provide hbase-site.xml from
the bundle so that
HBaseConfiguration can pick it up and ensure the right defaults.
The main problem I think is that there should not be any other HBase libs in
the class path since Hudi bundles already introduce the HBase classes. The
reason you see below is that the HBase class shaded in Hudi
(`org.apache.hudi.org.apache.hadoop.hbase.HBaseConfiguration`) reads the
`hbase-default.xml` from cdh HBase, which causes. This unfortunately happens
because `hbase-default.xml` is going to be under the root as well, without
isolation. @rbtrtr Could you share how you start the spark-shell? Wondering
if you have to pass in cdh HBase libs to Spark?
```
Caused by: java.lang.RuntimeException: hbase-default.xml file seems to be
for an older version of HBase (2.2.3.7.1.7.0-551), this version is 2.4.9
at
org.apache.hudi.org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:74)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
