Hi, thanks for you reply I noticed that I can check the classpath only on logs of successful jobs (oozie spark examples). In the application folders that refer to my custom workflow/spark jar I have only the few lines about the NoSuchMethodError. Anyway in the logs with the class path I found that the version of haddop-common is 2.7.0 according to the hadoop version used.
Checking the maven dependency tree I found that the spark core artifact I found that the version of spark I'm using (1.4.0) has a nested dependency on hadoop-client:2.2.0 which has a dependency on hadoop-mapreduce-client:2.2.0 which has actually the method org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main invoking the missing org.apache.hadoop.http.HttpConfig.setPolicy. I tried to exclude that dependency and submit the oozie job again and now I get a different error when I check the yarn log: INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:217) at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2624) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:497) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:281) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1496) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1493) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1426) I'm afraid that, as you said, I might be using a incompatible version of hadoop or oozie or spark. Everytime I fix an error I just get a new one and I believe that things should have been smother. 2015-07-21 14:35 GMT+02:00 Oussama Chougna <[email protected]>: > Hello, > Check which version of hadoop-common is on your classpath.You can check > this in the logs of the oozie action job. Each oozie action is executed via > a Mapper and thus submitted as a Hadoop job. > The log ouput looks like: > Files in current > dir:/yarn/nm/usercache/chougnaoa/appcache/application_1437465272054_0011/container_1437465272054_0011_01_000002/. > ====================== > File: .launch_container.sh.crc > File: launch_container.sh > File: .action.xml.crc > File: .container_tokens.crc > File: oozie-hadoop-utils-2.6.0-cdh5.4.2.oozie-4.1.0-cdh5.4.2.jar > File: .default_container_executor_session.sh.crc > File: oozie-sharelib-oozie-4.1.0-cdh5.4.2.jar > File: action.xml > File: job.xml > File: default_container_executor_session.sh > File: json-simple-1.1.jar > File: container_tokens > File: default_container_executor.sh > Dir: tmp > File: ftp.sh > File: .default_container_executor.sh.crc > File: .job.xml.crc > > Oozie Java/Map-Reduce/Pig action launcher-job configuration > ================================================================= > Workflow job id : 0000015-150721095438256-oozie-oozi-W > Workflow action id: 0000015-150721095438256-oozie-oozi-W@foobar > > Classpath : > ------------------------ > > /yarn/nm/usercache/chougnaoa/appcache/application_1437465272054_0011/container_1437465272054_0011_01_000002 > /etc/hadoop/conf.cloudera.yarn > /var/run/cloudera-scm-agent/process/293-yarn-NODEMANAGER > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-protobuf.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-common.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-format-javadoc.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-jackson.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common-2.6.0-cdh5.4.2.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-format.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-nfs-2.6.0-cdh5.4.2.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common.jar > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-nfs.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-annotations.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-hadoop.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-pig-bundle.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-scala_2.10.jar > > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common-2.6.0-cdh5.4.2-tests.jar....................... > I guess you are using an incompatible version with Yarn. > > > > > From: [email protected] > > Date: Tue, 21 Jul 2015 13:33:37 +0200 > > Subject: java.lang.NoSuchMethodError while running a spark job > > To: [email protected] > > > > Hi again! > > > > As I previously wrote in here ( > > > http://mail-archives.apache.org/mod_mbox/oozie-user/201507.mbox/%3CCALBGZ8o4n27S8w6fn3HFxfzJmZbA9Gsz71Ewg%2Br6XEFCZTFpPQ%40mail.gmail.com%3E > ) > > I'm running an oozie distro 4.2.0 built against hadoop 2.7.0. > > > > I'm trying to execute a fat spark jar that I developed using oozie spark > > action. The jar itself is perfectly running using a local/standalone/yarn > > master mode via normal spark-submit. > > > > My workflow is pretty simple > > > > <workflow-app xmlns='uri:oozie:workflow:0.5' name='SparkBatch'> > > <start to='spark-node' /> > > > > <action name='spark-node'> > > <spark xmlns="uri:oozie:spark-action:0.1"> > > <job-tracker>${jobTracker}</job-tracker> > > <name-node>${nameNode}</name-node> > > <master>${master}</master> > > <name>sparkbatch</name> > > <class>com.sparkBatch.storageArchitecture.App</class> > > > > <jar>${nameNode}/user/${wf:user()}/${batchAppFolder}/lib/${jarName}</jar> > > <!-- opts when using standalone/yarn --> > > <spark-opts>--num-executors 1 --driver-memory 2g > > --executor-memory 3g --executor-cores 4 --queue default</spark-opts> > > </spark> > > <ok to="end"/> > > <error to="fail"/> > > </action> > > <kill name="fail"> > > <message>Spark Job failed, error > > message[${wf:errorMessage(wf:lastErrorNode())}]</message> > > </kill> > > <end name="end"/> > > </workflow-app> > > > > I can submit the workflow on oozie server but the job status is stuck on > > RUNNING, no errors in the oozie job log. > > > > When I open the application console on hadoop I have this error: > > > > Application application_1437460801014_0011 failed 2 times due to AM > > Container for appattempt_1437460801014_0011_000002 exited with exitCode: > 1 > > For more detailed output, check application tracking page: > > > http://Matteos-MBP.local:8088/cluster/app/application_1437460801014_0011Then > , > > click on links to logs of each attempt. > > Diagnostics: Exception from container-launch. > > Container id: container_1437460801014_0011_02_000001 > > Exit code: 1 > > Stack trace: ExitCodeException exitCode=1: > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) > > at org.apache.hadoop.util.Shell.run(Shell.java:456) > > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) > > at > > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) > > at > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > > at > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Container exited with a non-zero exit code 1 > > Failing this attempt. Failing the application. > > > > > > and checking the yarn log via yarn logs -applicationId appId I have the > > following: > > > > 2015-07-20 22:51:36,794 INFO [main] > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for > > application appattempt_1437424593766_0001_000002 > > 2015-07-20 22:51:36,938 ERROR [main] > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting > MRAppMaster > > java.lang.NoSuchMethodError: > > > org.apache.hadoop.http.HttpConfig.setPolicy(Lorg/apache/hadoop/http/HttpConfig$Policy;)V > > at > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1364) > > 2015-07-20 22:51:36,940 INFO [Thread-1] > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a > > signal. Signaling RMCommunicator and JobHistoryEventHandler. > > > > From the last error it seems that some component is invoking a method > which > > has been removed since hadoop 2.3.0 and it is not found on the current > > version that I'm using (2.7.0) > > > > P.S I can correctly execute the oozie spark example which works with a > > local master > > > > I think it's still a problem of configuration/version between oozie and > > hadoop but after two days I can't figure out what is the problem. Did > > somebody already face this weird error? > > > > > > -- > > Matteo Remo Luzzi > > -- Matteo Remo Luzzi
