Hello together,
I am trying to run the Example-Jar from Giraph since at least 2 days.
Following I have tried:
1.
*
git clone https://git-wip-us.apache.org/repos/asf/giraph.git
*
mv giraph giraph_src; cd giraph_src
*
mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests clean package
*
echo export PATH=\$PATH:/usr/local/giraph/bin >> ~/.bashrc
*
echo export GIRAPH_HOME=/usr/local/giraph >> ~/.bashrc
*
echo export HADOOP_MAPRED_HOME=$HADOOP_PREFIX >> ~/.bashrc
*
source ~/.bashrc
*
jar=$GIRAPH_HOME/giraph-examples-1.1.0-hadoop-2.4.0-jar-with-dependencies.jar
*
runner=org.apache.giraph.GiraphRunner
*
computation=org.apache.giraph.examples.SimpleShortestPathsComputation
*
informat=org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
*
outformat=org.apache.giraph.io.formats.IdWithValueTextOutputFormat
*
hadoop jar $jar $runner $computation -vif $informat -vip
tiny_graph.txt -vof $outformat -op shortestpaths -w 1 -yj $jar
I have also put the Example-Jar to /usr/local/hadoop/share/hadoop and
restarted the single-cluster. But still the last command aborts and in
the userlogs you can see, that the main-class
org.apache.giraph.yarn.GiraphApplicationMaster couldnt be found.
How can u tell the taskrunner where to find the jar else?
2. I ve done the same compiling Giraph with Hadoop-version 2.2 and got
the same error.
3. Then I ve reverted the repository with git revert
ac93c3b6c5bce5f22b293b29df91663ca7d7ce63 described like on
http://mail-archives.apache.org/mod_mbox/giraph-user/201312.mbox/%3C1647021.5fbjhLDxPK@chronos7%3E.
But the revert got conflicts on quite 5 pom.xml-files - so I gave up,
while I am still mvn-newbie
4. Then I made a git fetch origin
ac93c3b6c5bce5f22b293b29df91663ca7d7ce63 described like on
http://stackoverflow.com/questions/3489173/how-to-clone-git-repository-with-specific-revision-changeset.
I got a clean state, but the mvn clean install|package went to errors
because of version-conflicts of many packages of the Apache Giraph Parent
This is the following output after executing the last command of 1. and 2.:
hadoop jar $jar $runner $computation -vif $informat -vip tiny_graph.txt
-vof $outformat -op shortestpaths -w 1 -yj $jar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hadoop-2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop-2.4.0/share/myLib/giraph-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop-2.4.0/share/myLib/giraph-examples-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/07/26 13:15:45 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
14/07/26 13:15:46 INFO utils.ConfigurationUtils: No edge input format
specified. Ensure your InputFormat does not require one.
14/07/26 13:15:46 INFO utils.ConfigurationUtils: No edge output format
specified. Ensure your OutputFormat does not require one.
14/07/26 13:15:46 INFO yarn.GiraphYarnClient: Final output path is:
hdfs://vmvis5:9000/user/hadoop/shortestpaths
14/07/26 13:15:46 INFO yarn.GiraphYarnClient: Running Client
14/07/26 13:15:46 INFO client.RMProxy: Connecting to ResourceManager at
vmvis5/141.71.31.187:8032
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Got node report from ASM
for, nodeId=vmvis5:48804, nodeAddress vmvis5:8042, nodeRackName
/default-rack, nodeNumContainers 0
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Defaulting per-task heap
size to 1024MB.
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Obtained new Application
ID: application_1406373194228_0003
14/07/26 13:15:47 INFO Configuration.deprecation: mapred.job.id is
deprecated. Instead, use mapreduce.job.id
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Set the environment for
the application master
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Environment for AM
:{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
14/07/26 13:15:47 INFO yarn.YarnUtils: Registered file in LocalResources
::
hdfs://vmvis5:9000/user/hadoop/giraph_yarn_jar_cache/application_1406373194228_0003/giraph-conf.xml
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: LIB JARS
:/usr/local/giraph/giraph-examples-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name .
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking .
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name /usr/local/hadoop
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking /usr/local/hadoop
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/etc/hadoop
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/etc/hadoop
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/share/hadoop/common/lib/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/share/hadoop/common/lib
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/share/hadoop/common/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/share/hadoop/common
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/share/hadoop/hdfs
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/share/hadoop/hdfs
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/share/hadoop/hdfs/lib/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/share/hadoop/hdfs/lib
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/share/hadoop/hdfs/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/share/hadoop/hdfs
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/share/hadoop/yarn/lib/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/share/hadoop/yarn/lib
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/share/hadoop/yarn/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/share/hadoop/yarn
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/share/hadoop/mapreduce/lib/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/share/hadoop/mapreduce/lib
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/share/hadoop/mapreduce/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/share/hadoop/mapreduce
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/contrib/capacity-scheduler/*.jar
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/contrib/capacity-scheduler
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/share/myLib/giraph-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/share/myLib/giraph-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name
/usr/local/hadoop/share/myLib/giraph-examples-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking
/usr/local/hadoop/share/myLib/giraph-examples-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar
14/07/26 13:15:47 INFO yarn.GiraphYarnClient:
ApplicationSumbissionContext for GiraphApplicationMaster launch
container is populated.
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/07/26 13:15:47 INFO impl.YarnClientImpl: Submitted application
application_1406373194228_0003
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Got new appId after
submission :application_1406373194228_0003
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
container request was submitted to ResourceManager for job: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation
14/07/26 13:15:48 INFO yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,85
secs
14/07/26 13:15:48 INFO yarn.GiraphYarnClient:
appattempt_1406373194228_0003_000001, State: ACCEPTED, Containers used: 1
14/07/26 13:15:51 ERROR yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
state, diagnostics show: Application application_1406373194228_0003
failed 2 times due to AM Container for
appattempt_1406373194228_0003_000002 exited with exitCode: 1 due to:
Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
14/07/26 13:15:51 INFO yarn.GiraphYarnClient: Cleaning up HDFS
distributed cache directory for Giraph job.
14/07/26 13:15:51 INFO yarn.GiraphYarnClient: Completed Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
running time: 0 minutes, 3 seconds.
<< END
I hope someone knows, what I did wrong. Maybe the Distribution-Cache in
Yarn is working differently and I have todo first something?
Thank you for your help!
Sincerely
Alex
