[jira] [Commented] (YARN-10314) YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars
[ https://issues.apache.org/jira/browse/YARN-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138090#comment-17138090 ] Hudson commented on YARN-10314: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18355 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18355/]) YARN-10314. YarnClient throws NoClassDefFoundError for (github: rev fc4ebb0499fe1095b87ff782c265e9afce154266) * (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml * (edit) hadoop-client-modules/hadoop-client-runtime/pom.xml > YarnClient throws NoClassDefFoundError for WebSocketException with only > shaded client jars > -- > > Key: YARN-10314 > URL: https://issues.apache.org/jira/browse/YARN-10314 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.3.0 >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Blocker > Fix For: 3.3.0 > > > After YARN-8778, with only shaded hadoop client jars in classpath Unable to > submit job. > CC: [~ayushtkn] confirmed the same. Hive 4.0 doesnot work due to this, shaded > client is necessary there to avoid guava jar's conflicts. > {noformat} > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/shaded/org/eclipse/jetty/websocket/api/WebSocketException > at > org.apache.hadoop.yarn.client.api.YarnClient.createYarnClient(YarnClient.java:92) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:109) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:153) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:130) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:109) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:102) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1545) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1541) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845) > at org.apache.hadoop.mapreduce.Job.connect(Job.java:1541) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1570) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1594) > at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.shaded.org.eclipse.jetty.websocket.api.WebSocketException > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > ... 16 more > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10314) YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars
[ https://issues.apache.org/jira/browse/YARN-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136382#comment-17136382 ] Masatake Iwasaki commented on YARN-10314: - Oops.. I needed conf dir on the classpath. {noformat} $ java -classpath etc/hadoop:share/hadoop/common/lib/*:share/hadoop/client/*:share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0-SNAPSHOT.jar org.apache.hadoop.examples.QuasiMonteCarlo 1 100 Number of Maps = 1 Samples per Map = 100 2020-06-16 16:18:08,250 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Wrote input for Map #0 Starting Job Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/org/eclipse/jetty/websocket/api/WebSocketException at org.apache.hadoop.yarn.client.api.YarnClient.createYarnClient(YarnClient.java:92) at org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:109) at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:153) at org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:130) at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:109) at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:102) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1545) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1541) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845) at org.apache.hadoop.mapreduce.Job.connect(Job.java:1541) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1570) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1594) at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:307) at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:360) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:368) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.org.eclipse.jetty.websocket.api.WebSocketException at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 19 more {noformat} > YarnClient throws NoClassDefFoundError for WebSocketException with only > shaded client jars > -- > > Key: YARN-10314 > URL: https://issues.apache.org/jira/browse/YARN-10314 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.3.0 >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Blocker > > After YARN-8778, with only shaded hadoop client jars in classpath Unable to > submit job. > CC: [~ayushtkn] confirmed the same. Hive 4.0 doesnot work due to this, shaded > client is necessary there to avoid guava jar's conflicts. > {noformat} > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/shaded/org/eclipse/jetty/websocket/api/WebSocketException > at > org.apache.hadoop.yarn.client.api.YarnClient.createYarnClient(YarnClient.java:92) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:109) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:153) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:130) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:109) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:102) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1545) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1541) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845) > at org.apache.hadoop.mapreduce.Job.connect(Job.java:1541) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1570) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1594) > at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.shaded.org.eclipse.jetty.websocket.ap
[jira] [Commented] (YARN-10314) YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars
[ https://issues.apache.org/jira/browse/YARN-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136379#comment-17136379 ] Masatake Iwasaki commented on YARN-10314: - [~vinayakumarb] could you elaborate the condition to reproduce the issue? I tried following on hadoop dist built by {{mvn package -DskipTests -Pdist -Pnative}} on trunk. {noformat} $ java -classpath share/hadoop/common/lib/*:share/hadoop/client/*:share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0-SNAPSHOT.jar org.apache.hadoop.examples.QuasiMonteCarlo 1 100 log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Number of Maps = 1 Samples per Map = 100 Wrote input for Map #0 Starting Job Job Finished in 1.458 seconds Estimated value of Pi is 3.2000 {noformat} > YarnClient throws NoClassDefFoundError for WebSocketException with only > shaded client jars > -- > > Key: YARN-10314 > URL: https://issues.apache.org/jira/browse/YARN-10314 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.3.0 >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Blocker > > After YARN-8778, with only shaded hadoop client jars in classpath Unable to > submit job. > CC: [~ayushtkn] confirmed the same. Hive 4.0 doesnot work due to this, shaded > client is necessary there to avoid guava jar's conflicts. > {noformat} > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/shaded/org/eclipse/jetty/websocket/api/WebSocketException > at > org.apache.hadoop.yarn.client.api.YarnClient.createYarnClient(YarnClient.java:92) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:109) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:153) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:130) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:109) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:102) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1545) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1541) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845) > at org.apache.hadoop.mapreduce.Job.connect(Job.java:1541) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1570) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1594) > at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.shaded.org.eclipse.jetty.websocket.api.WebSocketException > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > ... 16 more > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10314) YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars
[ https://issues.apache.org/jira/browse/YARN-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135669#comment-17135669 ] Vinayakumar B commented on YARN-10314: -- Simple wordcount example to confirm {noformat} export CLASSPATH=$HADOOP_HOME/share/hadoop/common/lib/*.jar:$HADOOP_HOME/share/hadoop/client/*.jar:$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples*.jar; java org.apache.hadoop.examples.WordCount /in /out {noformat} > YarnClient throws NoClassDefFoundError for WebSocketException with only > shaded client jars > -- > > Key: YARN-10314 > URL: https://issues.apache.org/jira/browse/YARN-10314 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.3.0 >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Blocker > > After YARN-8778, with only shaded hadoop client jars in classpath Unable to > submit job. > CC: [~ayushtkn] confirmed the same. Hive 4.0 doesnot work due to this, shaded > client is necessary there to avoid guava jar's conflicts. > {noformat} > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/shaded/org/eclipse/jetty/websocket/api/WebSocketException > at > org.apache.hadoop.yarn.client.api.YarnClient.createYarnClient(YarnClient.java:92) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:109) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:153) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:130) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:109) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:102) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1545) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1541) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845) > at org.apache.hadoop.mapreduce.Job.connect(Job.java:1541) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1570) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1594) > at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.shaded.org.eclipse.jetty.websocket.api.WebSocketException > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > ... 16 more > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org