To avoid dependency problems, make sure the dependency versions in your java project match with the versions on your hadoop cluster and also scope them as "provided". And there should be a log file for the job that executes the oozie action, please check your job history server for jobs with a name like: oozie:launcher:T=shell:W=Some workflow:A=some-action:ID=0000019-150721095438256-oozie-oozi-W
Cheers, Oussama Chougna > From: [email protected] > Date: Tue, 21 Jul 2015 15:50:12 +0200 > Subject: Re: java.lang.NoSuchMethodError while running a spark job > To: [email protected] > > Hi, thanks for you reply > > I noticed that I can check the classpath only on logs of successful jobs > (oozie spark examples). In the application folders that refer to my custom > workflow/spark jar I have only the few lines about the NoSuchMethodError. > Anyway in the logs with the class path I found that the version of > haddop-common is 2.7.0 according to the hadoop version used. > > Checking the maven dependency tree I found that the spark core artifact I > found that the version of spark I'm using (1.4.0) has a nested dependency > on hadoop-client:2.2.0 which has a dependency on > hadoop-mapreduce-client:2.2.0 which has actually the method > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main invoking the missing > org.apache.hadoop.http.HttpConfig.setPolicy. > > I tried to exclude that dependency and submit the oozie job again and now I > get a different error when I check the yarn log: > > INFO [main] org.apache.hadoop.service.AbstractService: Service > org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; > cause: java.lang.UnsupportedOperationException: Not implemented by the TFS > FileSystem implementation > java.lang.UnsupportedOperationException: Not implemented by the TFS > FileSystem implementation > at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:217) > at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2624) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:497) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:281) > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1496) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1493) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1426) > > I'm afraid that, as you said, I might be using a incompatible version of > hadoop or oozie or spark. Everytime I fix an error I just get a new one and > I believe that things should have been smother. > > 2015-07-21 14:35 GMT+02:00 Oussama Chougna <[email protected]>: > > > Hello, > > Check which version of hadoop-common is on your classpath.You can check > > this in the logs of the oozie action job. Each oozie action is executed via > > a Mapper and thus submitted as a Hadoop job. > > The log ouput looks like: > > Files in current > > dir:/yarn/nm/usercache/chougnaoa/appcache/application_1437465272054_0011/container_1437465272054_0011_01_000002/. > > ====================== > > File: .launch_container.sh.crc > > File: launch_container.sh > > File: .action.xml.crc > > File: .container_tokens.crc > > File: oozie-hadoop-utils-2.6.0-cdh5.4.2.oozie-4.1.0-cdh5.4.2.jar > > File: .default_container_executor_session.sh.crc > > File: oozie-sharelib-oozie-4.1.0-cdh5.4.2.jar > > File: action.xml > > File: job.xml > > File: default_container_executor_session.sh > > File: json-simple-1.1.jar > > File: container_tokens > > File: default_container_executor.sh > > Dir: tmp > > File: ftp.sh > > File: .default_container_executor.sh.crc > > File: .job.xml.crc > > > > Oozie Java/Map-Reduce/Pig action launcher-job configuration > > ================================================================= > > Workflow job id : 0000015-150721095438256-oozie-oozi-W > > Workflow action id: 0000015-150721095438256-oozie-oozi-W@foobar > > > > Classpath : > > ------------------------ > > > > /yarn/nm/usercache/chougnaoa/appcache/application_1437465272054_0011/container_1437465272054_0011_01_000002 > > /etc/hadoop/conf.cloudera.yarn > > /var/run/cloudera-scm-agent/process/293-yarn-NODEMANAGER > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-protobuf.jar > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-common.jar > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-format-javadoc.jar > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-jackson.jar > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common-2.6.0-cdh5.4.2.jar > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-format.jar > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-nfs-2.6.0-cdh5.4.2.jar > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-nfs.jar > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-annotations.jar > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-hadoop.jar > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-pig-bundle.jar > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-scala_2.10.jar > > > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common-2.6.0-cdh5.4.2-tests.jar....................... > > I guess you are using an incompatible version with Yarn. > > > > > > > > > From: [email protected] > > > Date: Tue, 21 Jul 2015 13:33:37 +0200 > > > Subject: java.lang.NoSuchMethodError while running a spark job > > > To: [email protected] > > > > > > Hi again! > > > > > > As I previously wrote in here ( > > > > > http://mail-archives.apache.org/mod_mbox/oozie-user/201507.mbox/%3CCALBGZ8o4n27S8w6fn3HFxfzJmZbA9Gsz71Ewg%2Br6XEFCZTFpPQ%40mail.gmail.com%3E > > ) > > > I'm running an oozie distro 4.2.0 built against hadoop 2.7.0. > > > > > > I'm trying to execute a fat spark jar that I developed using oozie spark > > > action. The jar itself is perfectly running using a local/standalone/yarn > > > master mode via normal spark-submit. > > > > > > My workflow is pretty simple > > > > > > <workflow-app xmlns='uri:oozie:workflow:0.5' name='SparkBatch'> > > > <start to='spark-node' /> > > > > > > <action name='spark-node'> > > > <spark xmlns="uri:oozie:spark-action:0.1"> > > > <job-tracker>${jobTracker}</job-tracker> > > > <name-node>${nameNode}</name-node> > > > <master>${master}</master> > > > <name>sparkbatch</name> > > > <class>com.sparkBatch.storageArchitecture.App</class> > > > > > > <jar>${nameNode}/user/${wf:user()}/${batchAppFolder}/lib/${jarName}</jar> > > > <!-- opts when using standalone/yarn --> > > > <spark-opts>--num-executors 1 --driver-memory 2g > > > --executor-memory 3g --executor-cores 4 --queue default</spark-opts> > > > </spark> > > > <ok to="end"/> > > > <error to="fail"/> > > > </action> > > > <kill name="fail"> > > > <message>Spark Job failed, error > > > message[${wf:errorMessage(wf:lastErrorNode())}]</message> > > > </kill> > > > <end name="end"/> > > > </workflow-app> > > > > > > I can submit the workflow on oozie server but the job status is stuck on > > > RUNNING, no errors in the oozie job log. > > > > > > When I open the application console on hadoop I have this error: > > > > > > Application application_1437460801014_0011 failed 2 times due to AM > > > Container for appattempt_1437460801014_0011_000002 exited with exitCode: > > 1 > > > For more detailed output, check application tracking page: > > > > > http://Matteos-MBP.local:8088/cluster/app/application_1437460801014_0011Then > > , > > > click on links to logs of each attempt. > > > Diagnostics: Exception from container-launch. > > > Container id: container_1437460801014_0011_02_000001 > > > Exit code: 1 > > > Stack trace: ExitCodeException exitCode=1: > > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) > > > at org.apache.hadoop.util.Shell.run(Shell.java:456) > > > at > > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) > > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) > > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > > at java.lang.Thread.run(Thread.java:745) > > > Container exited with a non-zero exit code 1 > > > Failing this attempt. Failing the application. > > > > > > > > > and checking the yarn log via yarn logs -applicationId appId I have the > > > following: > > > > > > 2015-07-20 22:51:36,794 INFO [main] > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for > > > application appattempt_1437424593766_0001_000002 > > > 2015-07-20 22:51:36,938 ERROR [main] > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting > > MRAppMaster > > > java.lang.NoSuchMethodError: > > > > > org.apache.hadoop.http.HttpConfig.setPolicy(Lorg/apache/hadoop/http/HttpConfig$Policy;)V > > > at > > > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1364) > > > 2015-07-20 22:51:36,940 INFO [Thread-1] > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a > > > signal. Signaling RMCommunicator and JobHistoryEventHandler. > > > > > > From the last error it seems that some component is invoking a method > > which > > > has been removed since hadoop 2.3.0 and it is not found on the current > > > version that I'm using (2.7.0) > > > > > > P.S I can correctly execute the oozie spark example which works with a > > > local master > > > > > > I think it's still a problem of configuration/version between oozie and > > > hadoop but after two days I can't figure out what is the problem. Did > > > somebody already face this weird error? > > > > > > > > > -- > > > Matteo Remo Luzzi > > > > > > > > -- > Matteo Remo Luzzi
