回复: flink 1.11 发布sql任务到yarn session报java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.util.ByteStringer

Fri, 14 Aug 2020 03:37:48 -0700 

终于找到了问题，原因是flink-dist-*.jar包中打入了高版本的protobuf-java(3.7.1)，高版本的protobuf-java中LiteralByteString是ByteString的私有内部类：

private static class LiteralByteString extends ByteString.LeafByteString {
  private static final long serialVersionUID = 1L;

  protected final byte[] bytes;

  /**
   * Creates a {@code LiteralByteString} backed by the given array, without 
copying.
   *
   * @param bytes array to wrap
   */
  LiteralByteString(byte[] bytes) {
    if (bytes == null) {
      throw new NullPointerException();
    }
    this.bytes = bytes;
  }

而HBase Connector(1.4.3) 
读取数据过程中初始化org.apache.hadoop.hbase.util.ByteStringer时调用了new 
LiteralByteString()，这样就无法找到该类，从而报了java.lang.NoClassDefFoundError: Could not 
initialize class org.apache.hadoop.hbase.util.ByteStringer。

解决方法：flink打包时去掉了protobuf-java(3.7.1)依赖，提交时将protobuf-java:2.5.0作为依赖即可。
________________________________
发件人: wind.fly....@outlook.com <wind.fly....@outlook.com>
发送时间: 2020年8月13日 10:09
收件人: user-zh@flink.apache.org <user-zh@flink.apache.org>
主题: flink 1.11 发布sql任务到yarn session报java.lang.NoClassDefFoundError: Could not 
initialize class org.apache.hadoop.hbase.util.ByteStringer

Hi, all:
         
本人试图将flink-sql-gateway(https://github.com/ververica/flink-sql-gateway)升级到1.11支持版本，将flink
 sql（用到hbase connector）提交到yarn session后运行时报:
        org.apache.hadoop.hbase.DoNotRetryIOException: 
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.hadoop.hbase.util.ByteStringer
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:248)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:221)
at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388)
at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
at 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.hadoop.hbase.util.ByteStringer
at 
org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:1053)
at 
org.apache.hadoop.hbase.protobuf.RequestConverter.buildScanRequest(RequestConverter.java:496)
at 
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:402)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:274)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
... 7 more
        
经过搜索怀疑可能是因为hbase-protobuf依赖的protobuf-java版本不对，但我怎么查看运行时jm、tm对应的classpath是什么样的，依赖了什么样的jar，希望给出分析思路或方法，谢谢。

回复