[
https://issues.apache.org/jira/browse/HDDS-6926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17607979#comment-17607979
]
Wei-Chiu Chuang commented on HDDS-6926:
---------------------------------------
Just to clarify, looks like spark-3.2.1-bin-hadoop3.2.tgz is actually built
with Hadoop 3.3 dependency. That is why there are "hadoop-client-api-3.3.1.jar"
and "hadoop-shaded-guava-1.1.1.jar" jars in the tarball.
I think the problem is that Ozone generates protobuf classes using the shaded
protobuf. But somehow Hadoop3OmTransport doesn't use them. And with Hadoop 3.3
dependency we should use Hadoop's ProtobufRpcEngine2 (which uses shaded
protobuf) not ProtobufRpcEngine.
> Cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message
> ----------------------------------------------------------------------
>
> Key: HDDS-6926
> URL: https://issues.apache.org/jira/browse/HDDS-6926
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Affects Versions: 1.2.1
> Environment: Ozone version:
> 1.2.1 and 1.3.0
> Spark version：
> 3.2.1
> Reporter: MLikeWater
> Priority: Major
>
> The test process is as follows:
> 1. Download and use the official Spark installation package:
> {code:java}
> spark-3.2.1-bin-hadoop3.2.tgz
> {code}
> 2. Place the ozone configuration file *ozone-site.xml* in ${SPARK_HOME}/conf
> and *ozone-filesystem-hadoop3-1.2.1.jar* in ${SPARK_HOME}/jars:
> {code:java}
> $ ll spark-3.2.1-bin-hadoop3.2/jars/hadoop-*
> rw-rr- 1 hadoop hadoop 19406393 Jun 17 11:27
> spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> rw-rr- 1 hadoop hadoop 31717292 Jun 17 11:27
> spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-runtime-3.3.1.jar
> rw-rr- 1 hadoop hadoop 3362359 Jun 17 11:27
> spark-3.2.1-bin-hadoop3.2/jars/hadoop-shaded-guava-1.1.1.jar
> rw-rr- 1 hadoop hadoop 56507 Jun 17 11:27
> spark-3.2.1-bin-hadoop3.2/jars/hadoop-yarn-server-web-proxy-3.3.1.jar
> \$ ll
> spark-3.2.1-bin-hadoop3.2/jars/ozone-filesystem-hadoop3-1.3.0-SNAPSHOT.jar
> rw-rr- 1 hadoop hadoop 61732949 Jun 18 15:15
> spark-3.2.1-bin-hadoop3.2/jars/ozone-filesystem-hadoop3-1.3.0-SNAPSHOT.jar
> \$ ll spark-3.2.1-bin-hadoop3.2/conf/ozone-site.xml
> rw-rr- 1 hadoop hadoop 10692 Jun 18 15:28
> spark-3.2.1-bin-hadoop3.2/conf/ozone-site.xml{code}
> However, Spark SQL fails to access Ozone data. The error is as follows:
> {code:java}
> 22/06/19 07:15:59 DEBUG FileSystem: Starting: Acquiring creator semaphore for
> ofs://cluster1/tgwarehouse
> 22/06/19 07:15:59 DEBUG FileSystem: Acquiring creator semaphore for
> ofs://cluster1/tgwarehouse: duration 0:00.000s
> 22/06/19 07:15:59 DEBUG Tracer: sampler.classes = ; loaded no samplers
> 22/06/19 07:15:59 DEBUG Tracer: span.receiver.classes = ; loaded no span
> receivers
> 22/06/19 07:15:59 DEBUG FileSystem: Starting: Creating FS
> ofs://cluster1/tgwarehouse
> 22/06/19 07:15:59 DEBUG FileSystem: Loading filesystems
> 22/06/19 07:15:59 DEBUG FileSystem: nullscan:// = class
> org.apache.hadoop.hive.ql.io.NullScanFileSystem from
> /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hive-exec-2.3.9-core.jar
> 22/06/19 07:15:59 DEBUG FileSystem: file:// = class
> org.apache.hadoop.fs.LocalFileSystem from
> /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: file:// = class
> org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem from
> /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hive-exec-2.3.9-core.jar
> 22/06/19 07:15:59 DEBUG FileSystem: viewfs:// = class
> org.apache.hadoop.fs.viewfs.ViewFileSystem from
> /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: har:// = class
> org.apache.hadoop.fs.HarFileSystem from
> /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: http:// = class
> org.apache.hadoop.fs.http.HttpFileSystem from
> /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: https:// = class
> org.apache.hadoop.fs.http.HttpsFileSystem from
> /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: hdfs:// = class
> org.apache.hadoop.hdfs.DistributedFileSystem from
> /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: webhdfs:// = class
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem from
> /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: swebhdfs:// = class
> org.apache.hadoop.hdfs.web.SWebHdfsFileSystem from
> /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: o3fs:// = class
> org.apache.hadoop.fs.ozone.OzoneFileSystem from
> /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/ozone-filesystem-hadoop3-1.3.0-SNAPSHOT.jar
> 22/06/19 07:15:59 DEBUG FileSystem: ofs:// = class
> org.apache.hadoop.fs.ozone.RootedOzoneFileSystem from
> /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/ozone-filesystem-hadoop3-1.3.0-SNAPSHOT.jar
> 22/06/19 07:15:59 DEBUG FileSystem: Looking for FS supporting ofs
> 22/06/19 07:15:59 DEBUG FileSystem: looking for configuration option
> fs.ofs.impl
> 22/06/19 07:15:59 DEBUG FileSystem: Filesystem ofs defined in configuration
> option
> 22/06/19 07:15:59 DEBUG FileSystem: FS for ofs is class
> org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
> 22/06/19 07:15:59 DEBUG Server: rpcKind=RPC_PROTOCOL_BUFFER,
> rpcRequestWrapperClass=class
> org.apache.hadoop.ipc.ProtobufRpcEngine2$RpcProtobufRequest,
> rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker@66933239
> 22/06/19 07:15:59 DEBUG Client: getting client out of cache:
> Client-3da530302dcb460d8e283786621c481f
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: RetryProxy: OM om1:
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest
> cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index
> to 1, nodeId: om2
> 22/06/19 07:15:59 DEBUG RetryInvocationHandler: java.lang.ClassCastException:
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest
> cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message, while
> invoking $Proxy26.submitRequest over
> nodeId=om1,nodeAddress=tg-local-bdworker-1.tg.mt.com:9862. Trying to failover
> immediately.
> java.lang.ClassCastException:
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest
> cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:123)
> at com.sun.proxy.$Proxy26.submitRequest(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy26.submitRequest(Unknown Source)
> at
> org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:282)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1440)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:235)
> at
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:247)
> at
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:114)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.<init>(BasicRootedOzoneClientAdapterImpl.java:179)
> at
> org.apache.hadoop.fs.ozone.RootedOzoneClientAdapterImpl.<init>(RootedOzoneClientAdapterImpl.java:51)
> at
> org.apache.hadoop.fs.ozone.RootedOzoneFileSystem.createAdapter(RootedOzoneFileSystem.java:92)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.initialize(BasicRootedOzoneFileSystem.java:149)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
> at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
> at
> org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$.hadoopFSsToAccess(HadoopFSDelegationTokenProvider.scala:176)
> at
> org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.obtainDelegationTokens(HadoopFSDelegationTokenProvider.scala:50)
> at
> org.apache.spark.deploy.security.HadoopDelegationTokenManager.$anonfun$obtainDelegationTokens$2(HadoopDelegationTokenManager.scala:164)
> at
> scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
> at scala.collection.Iterator.foreach(Iterator.scala:943)
> at scala.collection.Iterator.foreach$(Iterator.scala:943)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
> at
> scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214)
> at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
> at
> scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
> at
> org.apache.spark.deploy.security.HadoopDelegationTokenManager.org$apache$spark$deploy$security$HadoopDelegationTokenManager$$obtainDelegationTokens(HadoopDelegationTokenManager.scala:162)
> at
> org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anon$2.run(HadoopDelegationTokenManager.scala:148)
> at
> org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anon$2.run(HadoopDelegationTokenManager.scala:146)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> at
> org.apache.spark.deploy.security.HadoopDelegationTokenManager.obtainDelegationTokens(HadoopDelegationTokenManager.scala:146)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.$anonfun$start$1(CoarseGrainedSchedulerBackend.scala:555)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.$anonfun$start$1$adapted(CoarseGrainedSchedulerBackend.scala:549)
> at scala.Option.foreach(Option.scala:407)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.start(CoarseGrainedSchedulerBackend.scala:549)
> at
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend.start(KubernetesClusterSchedulerBackend.scala:95)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:220)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:581)
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
> at scala.Option.getOrElse(Option.scala:189)
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
> at
> org.apache.kyuubi.engine.spark.SparkSQLEngine$.createSpark(SparkSQLEngine.scala:186)
> at
> org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:277)
> at
> org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: Failing over OM from om1:0
> to om2:1
> 22/06/19 07:15:59 DEBUG Client: getting client out of cache:
> Client-3da530302dcb460d8e283786621c481f
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: RetryProxy: OM om2: null
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index
> to 2, nodeId: om3
> 22/06/19 07:15:59 INFO RetryInvocationHandler:
> java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over
> nodeId=om2,nodeAddress=tg-local-bdworker-2.tg.mt.com:9862 after 1 failover
> attempts. Trying to failover immediately.
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: Failing over OM from om2:1
> to om3:2
> 22/06/19 07:15:59 DEBUG Client: getting client out of cache:
> Client-3da530302dcb460d8e283786621c481f
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: RetryProxy: OM om3: null
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index
> to 0, nodeId: om1
> 22/06/19 07:15:59 INFO RetryInvocationHandler:
> java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over
> nodeId=om3,nodeAddress=tg-local-bdworker-3.tg.mt.com:9862 after 2 failover
> attempts. Trying to failover after sleeping for 2000ms.
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: Failing over OM from om3:2
> to om1:0
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: RetryProxy: OM om1: null
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index
> to 1, nodeId: om2
> 22/06/19 07:16:01 INFO RetryInvocationHandler:
> java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over
> nodeId=om1,nodeAddress=tg-local-bdworker-1.tg.mt.com:9862 after 3 failover
> attempts. Trying to failover immediately.
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: Failing over OM from om1:0
> to om2:1
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: RetryProxy: OM om2: null
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index
> to 2, nodeId: om3
> 22/06/19 07:16:01 INFO RetryInvocationHandler:
> java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over
> nodeId=om2,nodeAddress=tg-local-bdworker-2.tg.mt.com:9862 after 4 failover
> attempts. Trying to failover immediately.
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: Failing over OM from om2:1
> to om3:2
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: RetryProxy: OM om3: null
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index
> to 0, nodeId: om1
> 22/06/19 07:16:01 INFO RetryInvocationHandler:
> java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over
> nodeId=om3,nodeAddress=tg-local-bdworker-3.tg.mt.com:9862 after 5 failover
> attempts. Trying to failover after sleeping for 2000ms.
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: Failing over OM from om3:2
> to om1:0
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: RetryProxy: OM om1: null
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index
> to 1, nodeId: om2
> 22/06/19 07:16:03 INFO RetryInvocationHandler:
> java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over
> nodeId=om1,nodeAddress=tg-local-bdworker-1.tg.mt.com:9862 after 6 failover
> attempts. Trying to failover immediately.
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: Failing over OM from om1:0
> to om2:1
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: RetryProxy: OM om2: null
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index
> to 2, nodeId: om3
> 22/06/19 07:16:03 INFO RetryInvocationHandler:
> java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over
> nodeId=om2,nodeAddress=tg-local-bdworker-2.tg.mt.com:9862 after 7 failover
> attempts. Trying to failover immediately.
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: Failing over OM from om2:1
> to om3:2
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: RetryProxy: OM om3: null
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index
> to 0, nodeId: om1{code}
> If I use the following method to build Spark, the access is successful:
> {code:java}
> mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=3.2.2 -Phive -Phive-thriftserver
> -Pkubernetes -DskipTests clean package{code}
>
> Just use -Phadoop-2.7 to compile and ${SPARK_HOME}/jars contains the hadoop
> jar:
> {code:java}
> $ ll hadoop-*
> rw-rr- 1 hadoop hadoop 60199 Apr 21 11:15 hadoop-annotations-3.2.2.jar
> rw-rr- 1 hadoop hadoop 138914 Apr 21 11:15 hadoop-auth-3.2.2.jar
> rw-rr- 1 hadoop hadoop 44107 Apr 21 11:15 hadoop-client-3.2.2.jar
> rw-rr- 1 hadoop hadoop 4182881 Apr 21 11:15 hadoop-common-3.2.2.jar
> rw-rr- 1 hadoop hadoop 5139052 Apr 21 11:15 hadoop-hdfs-client-3.2.2.jar
> rw-rr- 1 hadoop hadoop 805828 Apr 21 11:15
> hadoop-mapreduce-client-common-3.2.2.jar
> rw-rr- 1 hadoop hadoop 1658798 Apr 21 11:15
> hadoop-mapreduce-client-core-3.2.2.jar
> rw-rr- 1 hadoop hadoop 85841 Apr 21 11:15
> hadoop-mapreduce-client-jobclient-3.2.2.jar
> rw-rr- 1 hadoop hadoop 3289927 Apr 21 11:15 hadoop-yarn-api-3.2.2.jar
> rw-rr- 1 hadoop hadoop 325110 Apr 21 11:15 hadoop-yarn-client-3.2.2.jar
> rw-rr- 1 hadoop hadoop 2919326 Apr 21 11:15 hadoop-yarn-common-3.2.2.jar
> rw-rr- 1 hadoop hadoop 80795 Apr 21 11:15
> hadoop-yarn-server-web-proxy-3.2.2.jar{code}
> Does not currently support Shaded Hadoop jars in Ozone?
> Thanks!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
