[
https://issues.apache.org/jira/browse/YARN-11030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454291#comment-17454291
]
Hiroyuki Adachi commented on YARN-11030:
----------------------------------------
[~prabhujoseph] Thank you for your reply. I checked that is a same issue.
Actually, it is applicable for not only jars on HDFS but also jars on local
filesystem.
> ClassNotFoundException when aux service class is loaded from customized
> classpath
> ---------------------------------------------------------------------------------
>
> Key: YARN-11030
> URL: https://issues.apache.org/jira/browse/YARN-11030
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 3.3.0, 3.3.1
> Reporter: Hiroyuki Adachi
> Priority: Minor
>
> NodeManager failed to load the aux service with ClassNotFoundException while
> loading the class from the customized classpath.
> {noformat}
> <configuration>
> <property name="yarn.nodemanager.aux-services" value="spark_shuffle"/>
> <property name="yarn.nodemanager.aux-services.spark_shuffle.class"
> value="org.apache.spark.network.yarn.YarnShuffleService"/>
> <property name="yarn.nodemanager.aux-services.spark_shuffle.classpath"
> value="/tmp/spark-3.1.2-yarn-shuffle.jar"/>
> <!-- Note that the spark_shuffle classpath is not included in
> HADOOP_CLASSPATH and yarn.application.classpath. -->
> </configuration> {noformat}
> {noformat}
> 2021-12-06 15:32:09,168 INFO org.apache.hadoop.util.ApplicationClassLoader:
> classpath: [file:/tmp/spark-3.1.2-yarn-shuffle.jar]
> 2021-12-06 15:32:09,168 INFO org.apache.hadoop.util.ApplicationClassLoader:
> system classes: [org.apache.spark.network.yarn.YarnShuffleService]
> 2021-12-06 15:32:09,169 INFO org.apache.hadoop.service.AbstractService:
> Service
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed
> in
> state INITED
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.ClassNotFoundException:
> org.apache.spark.network.yarn.YarnShuffleService
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.initAuxService(AuxServices.java:482)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:761)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:109)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:327)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:109)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:494)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:962)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1042)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.network.yarn.YarnShuffleService
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
> at
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.ja
> va:165)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.createAuxServiceFromLocalClasspath(AuxServices.java:242)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.createAuxService(AuxServices.java:271)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.initAuxService(AuxServices.java:452)
> ... 10 more
> 2021-12-06 15:32:09,172 INFO org.apache.hadoop.service.AbstractService:
> Service
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>
> failed in state INITED{noformat}
>
> YARN-9075 may cause this problem. The default system classes were changed by
> this patch.
> Before YARN-9075: isSystemClass() returns false since the system classes does
> not contain the aux service class itself, and the class will be loaded from
> the customized classpath.
> [https://github.com/apache/hadoop/blob/rel/release-3.3.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ApplicationClassLoader.java#L176]
> {noformat}
> 2021-12-06 15:50:21,332 INFO org.apache.hadoop.util.ApplicationClassLoader:
> classpath: [file:/tmp/spark-3.1.2-yarn-shuffle.jar]
> 2021-12-06 15:50:21,332 INFO org.apache.hadoop.util.ApplicationClassLoader:
> system classes: [java., javax.accessibility., -javax.activation.,
> javax.activity., j
> avax.annotation., javax.annotation.processing., javax.crypto.,
> javax.imageio., javax.jws., javax.lang.model., -javax.management.j2ee.,
> javax.management., javax.
> naming., javax.net., javax.print., javax.rmi., javax.script.,
> -javax.security.auth.message., javax.security.auth., javax.security.cert.,
> javax.security.sasl., j
> avax.sound., javax.sql., javax.swing., javax.tools., javax.transaction.,
> -javax.xml.registry., -javax.xml.rpc., javax.xml., org.w3c.dom.,
> org.xml.sax., org.apac
> he.commons.logging., org.apache.log4j., -org.apache.hadoop.hbase.,
> org.apache.hadoop., core-default.xml, hdfs-default.xml, mapred-default.xml,
> yarn-default.xml]
> {noformat}
> After YARN-9075: isSystemClass() returns true and the parent classloader will
> be used. It does not know the customized classpath and throws the
> ClassNotFoundException.
> [https://github.com/apache/hadoop/blob/rel/release-3.3.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ApplicationClassLoader.java#L189]
> {noformat}
> 2021-12-06 15:32:09,168 INFO org.apache.hadoop.util.ApplicationClassLoader:
> classpath: [file:/tmp/spark-3.1.2-yarn-shuffle.jar]
> 2021-12-06 15:32:09,168 INFO org.apache.hadoop.util.ApplicationClassLoader:
> system classes: [org.apache.spark.network.yarn.YarnShuffleService]{noformat}
>
> workaround: set dummy system-classes so that isSystemClass() returns false.
> {noformat}
> <configuration>
> <property
> name="yarn.nodemanager.aux-services.spark_shuffle.system-classes"
> value="dummy"/>
> </configuration>{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
