xi chaomin created HUDI-3983:
--------------------------------
Summary: ClassDefFound when using hudi-spark-bundle to write table
with hbase index
Key: HUDI-3983
URL: https://issues.apache.org/jira/browse/HUDI-3983
Project: Apache Hudi
Issue Type: Bug
Reporter: xi chaomin
I ran a spark job and encountered several ClassDefFound errors.
#
{code:java}
java.lang.NoClassDefFoundError:
org/apache/hudi/org/apache/hadoop/hbase/protobuf/generated/AuthenticationProtos$TokenIdentifier$Kind
at
org.apache.hudi.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:222)
at
org.apache.hudi.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:195)
at
org.apache.hudi.org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:395)
at
org.apache.hudi.org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
at
org.apache.hudi.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:108)
{code}
include org.apache.hbase:hbase-protocol in packaging/hudi-spark-bundle/pom.xml
can solve this error.
#
{code:java}
java.lang.ClassNotFoundException:
org.apache.hudi.org.apache.hbase.thirdparty.com.google.gson.GsonBuilder
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) {code}
include org.apache.hbase.thirdparty:hbase-shaded-gson n
packaging/hudi-spark-bundle/pom.xml can solve this error.
#
{code:java}
java.lang.ClassNotFoundException: Class
org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not
found {code}
There is a configuration in hbase-site.xml
{code:java}
<property>
<name>hbase.status.listener.class</name>
<value>org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener</value>
<description>
Implementation of the status listener with a multicast message.
</description>
</property> {code}
I set hbase.status.listener.class to
org.apache.hudi.org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener,
the ClassNotFoundException resolved, bug get another exception
{code:java}
org.apache.hudi.org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
Call to address=xxx:16020 failed on local exception:
org.apache.hudi.org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
Connection closed {code}
I remove the relocation related to hbase, the job succeed.
I don't know why ConnectionClosedException occurs when we use relocation.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
