limboys commented on issue #2904:
URL: https://github.com/apache/kyuubi/issues/2904#issuecomment-1415213895
This issue reappears in versions of Hadoop 3.3.4 and Spark-3.1-bin-hadoop3
when executing
```
base) [clvprd@hadoop-node3 jars]$ spark-submit --class
org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster
--driver-memory 1G --num-executors 3 --executor-memory 1G --executor-cores 1
/app/bigdata/hadoop/server/spark-3.3.1-bin-hadoop3/examples/jars/spark-examples_2.12-3.3.1.jar
100
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/shaded/javax/ws/rs/core/NoContentException
at
org.apache.hadoop.yarn.util.timeline.TimelineUtils.<clinit>(TimelineUtils.java:60)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:200)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:191)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1327)
at
org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1764)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
23/02/03 14:44:57 INFO ShutdownHookManager: Shutdown hook called
23/02/03 14:44:57 INFO ShutdownHookManager: Deleting directory
/tmp/spark-bed55fe1-7af2-42e2-a246-24f222abcdae
```
when the spark version is changed to spark-3.1-bin-hadoop3 is ok
```
spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master yarn \
> --deploy-mode cluster \
> --driver-memory 1G \
> --num-executors 3 \
> --executor-memory 1G \
> --executor-cores 1 \
>
/app/bigdata/hadoop/server/spark-3.2.3-bin-hadoop3.2/examples/jars/spark-examples_2.12-3.2.3.jar
100
2023-02-03 15:16:06,251 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2023-02-03 15:16:06,307 INFO client.DefaultNoHARMFailoverProxyProvider:
Connecting to ResourceManager at hadoop-node1/10.82.97.120:8032
2023-02-03 15:16:06,465 INFO client.AHSProxy: Connecting to Application
History server at hadoop-node1/10.82.97.120:10200
2023-02-03 15:16:06,584 INFO yarn.Client: Requesting a new application from
cluster with 3 NodeManagers
2023-02-03 15:16:07,144 INFO conf.Configuration: resource-types.xml not found
2023-02-03 15:16:07,144 INFO resource.ResourceUtils: Unable to find
'resource-types.xml'.
2023-02-03 15:16:07,161 INFO yarn.Client: Verifying our application has not
requested more than the maximum memory capability of the cluster (4096 MB per
container)
2023-02-03 15:16:07,161 INFO yarn.Client: Will allocate AM container, with
1408 MB memory including 384 MB overhead
2023-02-03 15:16:07,162 INFO yarn.Client: Setting up container launch
context for our AM
2023-02-03 15:16:07,162 INFO yarn.Client: Setting up the launch environment
for our AM container
2023-02-03 15:16:07,174 INFO yarn.Client: Preparing resources for our AM
container
....
2023-02-03 15:16:46,007 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: hadoop-node2
ApplicationMaster RPC port: 43052
queue: default
start time: 1675408575548
final status: **SUCCEEDED**
tracking URL:
http://hadoop-node1:8088/proxy/application_1675404773406_0001/
user: clvprd
2023-02-03 15:16:46,020 INFO util.ShutdownHookManager: Shutdown hook called
2023-02-03 15:16:46,021 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-fc581b74-8986-4937-9d2c-62258c7c1c31
2023-02-03 15:16:46,025 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-9cb0db4d-0932-43ab-bd2a-e08c1c4e6df2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
