Hello, Check which version of hadoop-common is on your classpath.You can check this in the logs of the oozie action job. Each oozie action is executed via a Mapper and thus submitted as a Hadoop job. The log ouput looks like: Files in current dir:/yarn/nm/usercache/chougnaoa/appcache/application_1437465272054_0011/container_1437465272054_0011_01_000002/. ====================== File: .launch_container.sh.crc File: launch_container.sh File: .action.xml.crc File: .container_tokens.crc File: oozie-hadoop-utils-2.6.0-cdh5.4.2.oozie-4.1.0-cdh5.4.2.jar File: .default_container_executor_session.sh.crc File: oozie-sharelib-oozie-4.1.0-cdh5.4.2.jar File: action.xml File: job.xml File: default_container_executor_session.sh File: json-simple-1.1.jar File: container_tokens File: default_container_executor.sh Dir: tmp File: ftp.sh File: .default_container_executor.sh.crc File: .job.xml.crc
Oozie Java/Map-Reduce/Pig action launcher-job configuration ================================================================= Workflow job id : 0000015-150721095438256-oozie-oozi-W Workflow action id: 0000015-150721095438256-oozie-oozi-W@foobar Classpath : ------------------------ /yarn/nm/usercache/chougnaoa/appcache/application_1437465272054_0011/container_1437465272054_0011_01_000002 /etc/hadoop/conf.cloudera.yarn /var/run/cloudera-scm-agent/process/293-yarn-NODEMANAGER /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-protobuf.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-common.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-format-javadoc.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-jackson.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common-2.6.0-cdh5.4.2.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-format.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-nfs-2.6.0-cdh5.4.2.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-nfs.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-annotations.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-hadoop.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-pig-bundle.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-scala_2.10.jar /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common-2.6.0-cdh5.4.2-tests.jar....................... I guess you are using an incompatible version with Yarn. > From: [email protected] > Date: Tue, 21 Jul 2015 13:33:37 +0200 > Subject: java.lang.NoSuchMethodError while running a spark job > To: [email protected] > > Hi again! > > As I previously wrote in here ( > http://mail-archives.apache.org/mod_mbox/oozie-user/201507.mbox/%3CCALBGZ8o4n27S8w6fn3HFxfzJmZbA9Gsz71Ewg%2Br6XEFCZTFpPQ%40mail.gmail.com%3E) > I'm running an oozie distro 4.2.0 built against hadoop 2.7.0. > > I'm trying to execute a fat spark jar that I developed using oozie spark > action. The jar itself is perfectly running using a local/standalone/yarn > master mode via normal spark-submit. > > My workflow is pretty simple > > <workflow-app xmlns='uri:oozie:workflow:0.5' name='SparkBatch'> > <start to='spark-node' /> > > <action name='spark-node'> > <spark xmlns="uri:oozie:spark-action:0.1"> > <job-tracker>${jobTracker}</job-tracker> > <name-node>${nameNode}</name-node> > <master>${master}</master> > <name>sparkbatch</name> > <class>com.sparkBatch.storageArchitecture.App</class> > > <jar>${nameNode}/user/${wf:user()}/${batchAppFolder}/lib/${jarName}</jar> > <!-- opts when using standalone/yarn --> > <spark-opts>--num-executors 1 --driver-memory 2g > --executor-memory 3g --executor-cores 4 --queue default</spark-opts> > </spark> > <ok to="end"/> > <error to="fail"/> > </action> > <kill name="fail"> > <message>Spark Job failed, error > message[${wf:errorMessage(wf:lastErrorNode())}]</message> > </kill> > <end name="end"/> > </workflow-app> > > I can submit the workflow on oozie server but the job status is stuck on > RUNNING, no errors in the oozie job log. > > When I open the application console on hadoop I have this error: > > Application application_1437460801014_0011 failed 2 times due to AM > Container for appattempt_1437460801014_0011_000002 exited with exitCode: 1 > For more detailed output, check application tracking page: > http://Matteos-MBP.local:8088/cluster/app/application_1437460801014_0011Then, > click on links to logs of each attempt. > Diagnostics: Exception from container-launch. > Container id: container_1437460801014_0011_02_000001 > Exit code: 1 > Stack trace: ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) > at org.apache.hadoop.util.Shell.run(Shell.java:456) > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Container exited with a non-zero exit code 1 > Failing this attempt. Failing the application. > > > and checking the yarn log via yarn logs -applicationId appId I have the > following: > > 2015-07-20 22:51:36,794 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for > application appattempt_1437424593766_0001_000002 > 2015-07-20 22:51:36,938 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster > java.lang.NoSuchMethodError: > org.apache.hadoop.http.HttpConfig.setPolicy(Lorg/apache/hadoop/http/HttpConfig$Policy;)V > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1364) > 2015-07-20 22:51:36,940 INFO [Thread-1] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a > signal. Signaling RMCommunicator and JobHistoryEventHandler. > > From the last error it seems that some component is invoking a method which > has been removed since hadoop 2.3.0 and it is not found on the current > version that I'm using (2.7.0) > > P.S I can correctly execute the oozie spark example which works with a > local master > > I think it's still a problem of configuration/version between oozie and > hadoop but after two days I can't figure out what is the problem. Did > somebody already face this weird error? > > > -- > Matteo Remo Luzzi
