Then do you know to solve this error? 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1415203231170_0002 failed 2 times due to AM Container for appattempt_1415203231170_0002_000002 exited with exitCode: 1 due to: Exception from container-launch. Container id: container_1415203231170_0002_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
On Wed, Nov 5, 2014 at 12:10 PM, Alessandro Negro <[email protected]> wrote: > Hi, > in the mean time I found the reason why it hangs for long time here: > > base path checking . > > > It scan all the directory tree starting from current directory, if you run > it into your home (and you have a lot of files and directories) it takes > ages to complete. > > I continue to got the same error. > > Regards, > Alessandro > > Il giorno 05/nov/2014, alle ore 18:33, Jing Fan <[email protected]> ha > scritto: > > I run with: > > hadoop jar > giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner > org.apache.giraph.examples.SimpleShortestPathsComputation -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat > -vip /user/jing/tiny_graph.txt -vof > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op > /user/jing/shortestpaths -w 1 -yj > giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar > > On Wed, Nov 5, 2014 at 11:25 AM, Alessandro Negro <[email protected]> > wrote: > >> This is my complete logs with the command I run: >> >> >> ale:~ />$HADOOP_INSTALL/bin/hadoop jar >> /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >> org.apache.giraph.GiraphRunner >> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >> -vip /tiny_graph.txt -vof >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /shortestpaths >> -w 1 >> 2014-11-05 13:39:30.235 java[3147:1903] Unable to load realm info from >> SCDynamicStore >> 14/11/05 13:39:30 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes where >> applicable >> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge input format >> specified. Ensure your InputFormat does not require one. >> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge output format >> specified. Ensure your OutputFormat does not require one. >> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Final output path is: >> hdfs://hadoop-master:9000/shortestpaths >> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Running Client >> 14/11/05 13:39:31 INFO client.RMProxy: Connecting to ResourceManager at / >> 0.0.0.0:8032 >> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Got node report from ASM >> for, nodeId=192.168.0.105:50658, nodeAddress 192.168.0.105:8042, >> nodeRackName /default-rack, nodeNumContainers 0 >> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Defaulting per-task heap >> size to 1024MB. >> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Obtained new Application >> ID: application_1415189069846_0004 >> 14/11/05 13:39:31 INFO Configuration.deprecation: mapred.job.id is >> deprecated. Instead, use mapreduce.job.id >> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Set the environment for the >> application master >> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Environment for AM >> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*} >> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... >> 14/11/05 13:39:31 INFO yarn.YarnUtils: Registered file in LocalResources >> :: >> hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415189069846_0004/giraph-conf.xml >> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: LIB JARS : >> 14/11/05 13:39:31 INFO yarn.YarnUtils: Class path name . >> 14/11/05 13:39:31 INFO yarn.YarnUtils: base path checking . >> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name null >> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking null >> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >> /opt/yarn/hadoop-2.5.1/etc/hadoop >> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >> /opt/yarn/hadoop-2.5.1/etc/hadoop >> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/* >> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib >> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >> /opt/yarn/hadoop-2.5.1/share/hadoop/common/* >> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >> /opt/yarn/hadoop-2.5.1/share/hadoop/common >> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs >> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs >> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/* >> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib >> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/* >> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs >> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/* >> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib >> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/* >> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn >> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/* >> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib >> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/* >> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce >> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >> /contrib/capacity-scheduler/*.jar >> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >> /contrib/capacity-scheduler >> 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: >> ApplicationSumbissionContext for GiraphApplicationMaster launch container >> is populated. >> 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: Submitting application to >> ASM >> 14/11/05 14:42:57 INFO impl.YarnClientImpl: Submitted application >> application_1415189069846_0004 >> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Got new appId after >> submission :application_1415189069846_0004 >> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: GiraphApplicationMaster >> container request was submitted to ResourceManager for job: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation >> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,82 >> secs >> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: >> appattempt_1415189069846_0004_000001, State: ACCEPTED, Containers used: 1 >> 14/11/05 14:43:01 ERROR yarn.GiraphYarnClient: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED >> state, diagnostics show: Application application_1415189069846_0004 failed >> 2 times due to AM Container for appattempt_1415189069846_0004_000002 exited >> with exitCode: 1 due to: Exception from container-launch: >> ExitCodeException exitCode=1: >> ExitCodeException exitCode=1: >> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >> at org.apache.hadoop.util.Shell.run(Shell.java:455) >> at >> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) >> at >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> >> >> Container exited with a non-zero exit code 1 >> .Failing this attempt.. Failing the application. >> 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Cleaning up HDFS >> distributed cache directory for Giraph job. >> 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Completed Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total >> running time: 0 minutes, 3 seconds. >> >> >> Il giorno 05/nov/2014, alle ore 18:17, Tripti Singh <[email protected]> >> ha scritto: >> >> I faced the similar problem where the workflow just hangs. >> Can u see in ur logs if there is a task with id 0. >> I made a local fix and I could give u the patch if u see a similar >> problem and ur workflow is unable to find the zookeeper manager file and >> repeats the following message: >> For task xyz, got file 'null' >> >> Thanks, >> Tripti >> Sent from my iPhone >> >> On 05-Nov-2014, at 10:40 pm, "Alessandro Negro" <[email protected]> >> wrote: >> >> Hi, >> have you seen the hadoop logs? >> Could you write here how are you running the command? >> >> Thanks, >> Alessandro >> >> Il giorno 05/nov/2014, alle ore 18:04, Jing Fan <[email protected]> >> ha scritto: >> >> Yeah, I add -yj option. The result is the same. Still error here. >> >> On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negro <[email protected]> >> wrote: >> >>> Hi Jing, >>> I’m facing the same problem. Do you check the hadoop logs? In my case I >>> got: >>> >>> Could not find or load main class >>> org.apache.giraph.yarn.GiraphApplicationMaster >>> >>> Moreover I don’t know why my application hang for a long time here: >>> >>> 14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking . >>> >>> And then continue failing with your error into the output and the >>> other into the hadoop logs. >>> >>> Did you test adding this? >>> >>> -yj >>> /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>> >>> Thanks, >>> Alessandro >>> >>> >>> Il giorno 05/nov/2014, alle ore 17:32, Jing Fan <[email protected]> >>> ha scritto: >>> >>> Hi, >>> >>> I install the Giraph with Yarn. >>> But when I run the program with following command: >>> >>> hadoop jar >>> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >>> org.apache.giraph.GiraphRunner >>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>> -vip /user/j/tiny_graph.txt -vof >>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >>> /user/j/shortestpaths -w 1 >>> >>> The program exited with error: >>> >>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.83 >>> secs >>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: >>> appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers used: 0 >>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 4.84 >>> secs >>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: >>> appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers used: 0 >>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 8.85 >>> secs >>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: >>> appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers used: 0 >>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 12.86 >>> secs >>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: >>> appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers used: 0 >>> 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED >>> state, diagnostics show: Application application_1415203231170_0002 failed >>> 2 times due to AM Container for appattempt_1415203231170_0002_000002 exited >>> with exitCode: 1 due to: Exception from container-launch. >>> Container id: container_1415203231170_0002_02_000001 >>> Exit code: 1 >>> Stack trace: ExitCodeException exitCode=1: >>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >>> at org.apache.hadoop.util.Shell.run(Shell.java:455) >>> at >>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> Anyone can help? >>> >>> >>> >> >> >> > >
