Hi, yea my logs shows that error. I follow this example: http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%[email protected]%3E
without success. Then I try to add all jars I can from giraph adding the following lines into the hadoop-env.sh: export MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB when I run the application now I see this: 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/myLib/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/myLib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/lib/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph/lib But again the same error. What I think is that there is no problem in the library but while is starting up. I saw this piece in the logs: 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated. 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application application_1415264041937_0005 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1415264041937_0005 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83 secs That show that it is able to load GiraphApplicationMaster but fail during the startup. Where can I find more logs? Thanks, Alessandro Il giorno 06/nov/2014, alle ore 06:56, Tripti Singh <[email protected]> ha scritto: > Can u run the following command to find container logs and post it here: > > yarn logs -applicationId application_1415203231170_0002 > > If u find this error: > Could not find or load main class > org.apache.giraph.yarn.GiraphApplicationMaster > It may just be due to non-availablity of the yarn jars > Thanks, > Tripti > Tech Yahoo > > From: Jing Fan <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Thursday, November 6, 2014 at 1:56 AM > To: "[email protected]" <[email protected]> > Subject: Re: Help with Giraph on Yarn > > Then do you know to solve this error? > > 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED > state, diagnostics show: Application application_1415203231170_0002 failed 2 > times due to AM Container for appattempt_1415203231170_0002_000002 exited > with exitCode: 1 due to: Exception from container-launch. > Container id: container_1415203231170_0002_02_000001 > Exit code: 1 > Stack trace: ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > On Wed, Nov 5, 2014 at 12:10 PM, Alessandro Negro <[email protected]> wrote: >> Hi, >> in the mean time I found the reason why it hangs for long time here: >> >> base path checking . >> >> >> It scan all the directory tree starting from current directory, if you run >> it into your home (and you have a lot of files and directories) it takes >> ages to complete. >> >> I continue to got the same error. >> >> Regards, >> Alessandro >> >> Il giorno 05/nov/2014, alle ore 18:33, Jing Fan <[email protected]> ha >> scritto: >> >>> I run with: >>> >>> hadoop jar >>> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >>> org.apache.giraph.GiraphRunner >>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>> -vip /user/jing/tiny_graph.txt -vof >>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >>> /user/jing/shortestpaths -w 1 -yj >>> giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >>> >>> On Wed, Nov 5, 2014 at 11:25 AM, Alessandro Negro <[email protected]> >>> wrote: >>>> This is my complete logs with the command I run: >>>> >>>> >>>> ale:~ />$HADOOP_INSTALL/bin/hadoop jar >>>> /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>>> org.apache.giraph.GiraphRunner >>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>>> -vip /tiny_graph.txt -vof >>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >>>> /shortestpaths -w 1 >>>> 2014-11-05 13:39:30.235 java[3147:1903] Unable to load realm info from >>>> SCDynamicStore >>>> 14/11/05 13:39:30 WARN util.NativeCodeLoader: Unable to load native-hadoop >>>> library for your platform... using builtin-java classes where applicable >>>> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge input format >>>> specified. Ensure your InputFormat does not require one. >>>> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge output format >>>> specified. Ensure your OutputFormat does not require one. >>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Final output path is: >>>> hdfs://hadoop-master:9000/shortestpaths >>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Running Client >>>> 14/11/05 13:39:31 INFO client.RMProxy: Connecting to ResourceManager at >>>> /0.0.0.0:8032 >>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Got node report from ASM >>>> for, nodeId=192.168.0.105:50658, nodeAddress 192.168.0.105:8042, >>>> nodeRackName /default-rack, nodeNumContainers 0 >>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Defaulting per-task heap >>>> size to 1024MB. >>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Obtained new Application ID: >>>> application_1415189069846_0004 >>>> 14/11/05 13:39:31 INFO Configuration.deprecation: mapred.job.id is >>>> deprecated. Instead, use mapreduce.job.id >>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Set the environment for the >>>> application master >>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Environment for AM >>>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*} >>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... >>>> 14/11/05 13:39:31 INFO yarn.YarnUtils: Registered file in LocalResources >>>> :: >>>> hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415189069846_0004/giraph-conf.xml >>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: LIB JARS : >>>> 14/11/05 13:39:31 INFO yarn.YarnUtils: Class path name . >>>> 14/11/05 13:39:31 INFO yarn.YarnUtils: base path checking . >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name null >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking null >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>> /opt/yarn/hadoop-2.5.1/etc/hadoop >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>> /opt/yarn/hadoop-2.5.1/etc/hadoop >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/* >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/common/* >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/common >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/* >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/* >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/* >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/* >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/* >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/* >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>> /contrib/capacity-scheduler/*.jar >>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>> /contrib/capacity-scheduler >>>> 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext >>>> for GiraphApplicationMaster launch container is populated. >>>> 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: Submitting application to ASM >>>> 14/11/05 14:42:57 INFO impl.YarnClientImpl: Submitted application >>>> application_1415189069846_0004 >>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Got new appId after >>>> submission :application_1415189069846_0004 >>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: GiraphApplicationMaster >>>> container request was submitted to ResourceManager for job: Giraph: >>>> org.apache.giraph.examples.SimpleShortestPathsComputation >>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Giraph: >>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,82 >>>> secs >>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: >>>> appattempt_1415189069846_0004_000001, State: ACCEPTED, Containers used: 1 >>>> 14/11/05 14:43:01 ERROR yarn.GiraphYarnClient: Giraph: >>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED >>>> state, diagnostics show: Application application_1415189069846_0004 failed >>>> 2 times due to AM Container for appattempt_1415189069846_0004_000002 >>>> exited with exitCode: 1 due to: Exception from container-launch: >>>> ExitCodeException exitCode=1: >>>> ExitCodeException exitCode=1: >>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >>>> at org.apache.hadoop.util.Shell.run(Shell.java:455) >>>> at >>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:744) >>>> >>>> >>>> Container exited with a non-zero exit code 1 >>>> .Failing this attempt.. Failing the application. >>>> 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed >>>> cache directory for Giraph job. >>>> 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Completed Giraph: >>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total >>>> running time: 0 minutes, 3 seconds. >>>> >>>> >>>> Il giorno 05/nov/2014, alle ore 18:17, Tripti Singh <[email protected]> >>>> ha scritto: >>>> >>>>> I faced the similar problem where the workflow just hangs. >>>>> Can u see in ur logs if there is a task with id 0. >>>>> I made a local fix and I could give u the patch if u see a similar >>>>> problem and ur workflow is unable to find the zookeeper manager file and >>>>> repeats the following message: >>>>> For task xyz, got file 'null' >>>>> >>>>> Thanks, >>>>> Tripti >>>>> Sent from my iPhone >>>>> >>>>> On 05-Nov-2014, at 10:40 pm, "Alessandro Negro" <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> have you seen the hadoop logs? >>>>>> Could you write here how are you running the command? >>>>>> >>>>>> Thanks, >>>>>> Alessandro >>>>>> >>>>>> Il giorno 05/nov/2014, alle ore 18:04, Jing Fan <[email protected]> ha >>>>>> scritto: >>>>>> >>>>>>> Yeah, I add -yj option. The result is the same. Still error here. >>>>>>> >>>>>>> On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negro <[email protected]> >>>>>>> wrote: >>>>>>>> Hi Jing, >>>>>>>> I’m facing the same problem. Do you check the hadoop logs? In my case >>>>>>>> I got: >>>>>>>> Could not find or load main class >>>>>>>> org.apache.giraph.yarn.GiraphApplicationMaster >>>>>>>> Moreover I don’t know why my application hang for a long time here: >>>>>>>> >>>>>>>> 14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking . >>>>>>>> >>>>>>>> And then continue failing with your error into the output and the >>>>>>>> other into the hadoop logs. >>>>>>>> >>>>>>>> Did you test adding this? >>>>>>>> >>>>>>>> -yj >>>>>>>> /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Alessandro >>>>>>>> >>>>>>>> >>>>>>>> Il giorno 05/nov/2014, alle ore 17:32, Jing Fan <[email protected]> >>>>>>>> ha scritto: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I install the Giraph with Yarn. >>>>>>>>> But when I run the program with following command: >>>>>>>>> >>>>>>>>> hadoop jar >>>>>>>>> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >>>>>>>>> org.apache.giraph.GiraphRunner >>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>>>>>>>> -vip /user/j/tiny_graph.txt -vof >>>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >>>>>>>>> /user/j/shortestpaths -w 1 >>>>>>>>> >>>>>>>>> The program exited with error: >>>>>>>>> >>>>>>>>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph: >>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: >>>>>>>>> 0.83 secs >>>>>>>>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: >>>>>>>>> appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers >>>>>>>>> used: 0 >>>>>>>>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph: >>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: >>>>>>>>> 4.84 secs >>>>>>>>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: >>>>>>>>> appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers >>>>>>>>> used: 0 >>>>>>>>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph: >>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: >>>>>>>>> 8.85 secs >>>>>>>>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: >>>>>>>>> appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers >>>>>>>>> used: 0 >>>>>>>>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph: >>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: >>>>>>>>> 12.86 secs >>>>>>>>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: >>>>>>>>> appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers >>>>>>>>> used: 0 >>>>>>>>> 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: >>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports >>>>>>>>> FAILED state, diagnostics show: Application >>>>>>>>> application_1415203231170_0002 failed 2 times due to AM Container for >>>>>>>>> appattempt_1415203231170_0002_000002 exited with exitCode: 1 due to: >>>>>>>>> Exception from container-launch. >>>>>>>>> Container id: container_1415203231170_0002_02_000001 >>>>>>>>> Exit code: 1 >>>>>>>>> Stack trace: ExitCodeException exitCode=1: >>>>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >>>>>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:455) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) >>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> >>>>>>>>> >>>>>>>>> Anyone can help? >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>> >> >
