[ 
https://issues.apache.org/jira/browse/YARN-9967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17053104#comment-17053104
 ] 

Tarun Parimi commented on YARN-9967:
------------------------------------

Hi [~snemeth], 
You can take it over. 
Thanks.

> Fix NodeManager failing to start when Hdfs Auxillary Jar is set
> ---------------------------------------------------------------
>
>                 Key: YARN-9967
>                 URL: https://issues.apache.org/jira/browse/YARN-9967
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: auxservices, nodemanager
>    Affects Versions: 3.3.0
>            Reporter: Prabhu Joseph
>            Assignee: Tarun Parimi
>            Priority: Major
>
> Loading an auxiliary jar from a Hdfs location on a node manager fails with 
> ClassNotFound Exception
> {code:java}
> 2019-11-08 03:59:49,256 INFO org.apache.hadoop.util.ApplicationClassLoader: 
> classpath: []
> 2019-11-08 03:59:49,256 INFO org.apache.hadoop.util.ApplicationClassLoader: 
> system classes: [java., javax.accessibility., javax.activation., 
> javax.activity., javax.annotation., javax.annotation.processing., 
> javax.crypto., javax.imageio., javax.jws., javax.lang.model., 
> -javax.management.j2ee., javax.management., javax.naming., javax.net., 
> javax.print., javax.rmi., javax.script., -javax.security.auth.message., 
> javax.security.auth., javax.security.cert., javax.security.sasl., 
> javax.sound., javax.sql., javax.swing., javax.tools., javax.transaction., 
> -javax.xml.registry., -javax.xml.rpc., javax.xml., org.w3c.dom., 
> org.xml.sax., org.apache.commons.logging., org.apache.log4j., 
> -org.apache.hadoop.hbase., org.apache.hadoop., core-default.xml, 
> hdfs-default.xml, mapred-default.xml, yarn-default.xml]
> 2019-11-08 03:59:49,257 INFO org.apache.hadoop.service.AbstractService: 
> Service 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed 
> in state INITED
> java.lang.ClassNotFoundException: org.apache.auxtest.AuxServiceFromHDFS
>       at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>       at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>       at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>       at 
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
>       at 
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
>       at java.lang.Class.forName0(Native Method)
>       at java.lang.Class.forName(Class.java:348)
>       at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
>       at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:270)
>       at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
>       at 
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
>       at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:321)
>       at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
>       at 
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
>       at 
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:478)
>       at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
>       at 
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:936)
>       at 
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1016)
> {code}
> *Repro:*
> {code:java}
> 1. Prepare a custom auxiliary service jar and place it on hdfs
> [hdfs@yarndocker-1 yarn]$ cat TestShuffleHandler2.java 
> package org;
> import org.apache.hadoop.yarn.server.api.AuxiliaryService;
> import org.apache.hadoop.yarn.server.api.ApplicationInitializationContext;
> import org.apache.hadoop.yarn.server.api.ApplicationTerminationContext;
> import java.nio.ByteBuffer;
> public class TestShuffleHandler2 extends AuxiliaryService {
>     public static final String MAPREDUCE_TEST_SHUFFLE_SERVICEID = 
> "test_shuffle2";
>     public TestShuffleHandler2() {
>       super("testshuffle2");
>     }
>     @Override
>     public void initializeApplication(ApplicationInitializationContext 
> context) {
>     }
>     @Override
>     public void stopApplication(ApplicationTerminationContext context) {
>     }
>     @Override
>     public synchronized ByteBuffer getMetaData() {
>       return ByteBuffer.allocate(0); 
>     }
>   }
>   
> [hdfs@yarndocker-1 yarn]$ javac -d . -cp `hadoop classpath` 
> TestShuffleHandler2.java 
> [hdfs@yarndocker-1 yarn]$ jar cvf auxhdfs.jar org/
> [hdfs@yarndocker-1 mapreduce]$ hadoop fs -mkdir /AUX
> [hdfs@yarndocker-1 mapreduce]$ hadoop fs -put /tmp/auxhdfs.jar /AUX
> [hdfs@yarndocker-1 mapreduce]$ hadoop fs -chmod 777 /AUX
> [hdfs@yarndocker-1 mapreduce]$ hadoop fs -chmod 600 /AUX/auxhdfs.jar
> [hdfs@yarndocker-1 mapreduce]$ hadoop fs -chown -R yarn:hadoop  /AUX
> 2. Configure YARN NodeManager (yarn-site.xml) to pick from Hdfs
>       <property>
>         <name>yarn.nodemanager.aux-services</name>
>         <value>auxhdfs</value>
>       </property>
>       <property>
>         <name>yarn.nodemanager.aux-services.auxhdfs.class&lt;/name>
>         <value>org.TestShuffleHandler2</value>
>       </property>
>                 
>       <property>
>         <name>yarn.nodemanager.aux-services.auxhdfs.remote-classpath</name>
>         <value>/AUX/auxhdfs.jar</value>
>       </property>
> {code}
>  
>  YARN-9965 issue is applicable for branch-3.2 and trunk (with 
> yarn.nodemanager.aux-services.auxhdfs.system-classes configured) where NM 
> runs fine on first time and not on subsequent restart.
> This issue is applicable for trunk where NM fails to start any attempt 
> without yarn.nodemanager.aux-services.auxhdfs.system-classes. This was an 
> optional config in branch-3.2. YARN-9075 has set the configured custom class 
> as a system class by default which caused the ClassNotFoundException as this 
> class won't be in system class loader.
> AuxServices#getSystemClasses has to return Null if not configured 
> {code:java}
>   private static String[] getSystemClasses(AuxServiceRecord service, String
>       className) {
>     AuxServiceConfiguration serviceConf =
>         service.getConfiguration();
>     if (serviceConf != null && serviceConf.getProperty(SYSTEM_CLASSES) != 
> null) {
>       return StringUtils.split(serviceConf.getProperty(SYSTEM_CLASSES));
>     }
>     return null;
>   }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to