Hi,
I am trying to launch a remote mapreduce job, but I get these 2 errors
below. The first error is the output of the execution. The second error
happens in the tasks.
Looking to the error, and my code, it seems that the job couldn’t load
|yarn.application.classpath| defined in the code. The path that I define
there is set in the |.bashrc| file (I show it below).
It seems that this error is related to the env vars set in my job, but
this vars are set in the |.bashrc|. Any help to understand how I fix
this error?
*Output of execution:*
|For more detailed output, check application tracking
page:http://hadoop-coc-1:8088/proxy/application_1424867716027_0002/Then, click
on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1424867716027_0002_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
|
*Task error*
|
Error: Could not find or load main class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster
|
*My code*
|String tempPath = "temp-" + System.currentTimeMillis();
for (String host : remoteHosts) {
Configuration conf = job.getConfiguration();
// this should be like defined in your yarn-site.xml
conf.set("yarn.resourcemanager.address", host + ":8032");
// framework is now "yarn", should be defined like this in
mapred-site.xm
conf.set("mapreduce.framework.name", "yarn");
conf.set("yarn.application.classpath ","$HADOOP_CONF_DIR,
$HADOOP_HOME/share/hadoop/common/*, $HADOOP_HOME/share/hadoop/common/lib/*, "
+ "$HADOOP_HOME/share/hadoop/hdfs/*,
$HADOOP_HOME/share/hadoop/hdfs/lib/*,"
+ "$HADOOP_HOME/share/hadoop/mapreduce/*,
$HADOOP_HOME/share/hadoop/mapreduce/lib/*, $HADOOP_HOME/share/hadoop/yarn/*,"
+ "$HADOOP_HOME/share/hadoop/yarn/lib/*, $HADOOP_HOME/*"
);
// like defined in hdfs-site.xml
conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
for (Path inputPath : inputPaths)
// FileInputFormat.addInputPath(job, new
Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
FileInputFormat.addInputPath(job, new
Path(inputPath.toString()));
// FileOutputFormat.setOutputPath(job, new
Path(conf.get("fs.defaultFS") + "/" + tempPath));
FileOutputFormat.setOutputPath(job, new Path(tempPath));
job.waitForCompletion(true);
}
|
*My .bashrc*
| JAVA_HOME=/usr/lib/jvm/java-8-oracle
HADOOP_HOME=$HOME/Programs/hadoop
HADOOP_CONF_HOME=$HADOOP_HOME/etc/hadoop
export HADOOP_MANAGER=$HOME/Programs/hadoop-mapreduce-manager-python
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
|
Thanks,
