Hi, I forgot to say that I added also this line: export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB
Il giorno 21/nov/2014, alle ore 10:34, Alessandro Negro <[email protected]> ha scritto: > Hi Das, > this is my configuration of hadoop: > > yarn 1029 0,1 1,2 3499572 209560 s001 S 10:29am 0:06.04 > /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java > -Dproc_resourcemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log > > -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml > -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log > > -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1 > -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath > /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/rm-config/log4j.properties > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager > yarn 1073 0,1 1,0 3347132 174644 s001 S 10:29am 0:04.77 > /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java > -Dproc_nodemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -server > -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1 > -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath > /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/nm-config/log4j.properties > org.apache.hadoop.yarn.server.nodemanager.NodeManager > > So I run it as yarn user in this way: > > sudo -u yarn …. > > Moreover I have just notice that I added the following lines into the > hadoop-env.sh > > hadoop-env.sh:export > MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar > > Accordingly to this other thread: > > http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%[email protected]%3E > > Let me know, > Alessandro > > > Il giorno 20/nov/2014, alle ore 23:51, D Adams <[email protected]> ha > scritto: > >> --f46d044283ecf837440508522939 >> Content-Type: text/plain; charset=UTF-8 >> Content-Transfer-Encoding: quoted-printable >> >> Ok, in the following, I use bin/yarn. Doesn't seem like much has changed: >> >> hduser@Roosevelt:/usr/local/hadoop$ ./hduser_jobs.sh >> Running hduser script ... >> bin/yarn jar >> /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for= >> -hadoop-2.5.1-jar-with-dependencies.jar >> org.apache.giraph.GiraphRunner >> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >> -vip /user/hduser/input/tiny_graph.txt -vof >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >> /user/hduser/output/shortestpaths -w 1 -yj >> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >> 14/11/18 05:36:05 WARN util.NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge input format >> specified. Ensure your InputFormat does not require one. >> 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge output format >> specified. Ensure your OutputFormat does not require one. >> 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Final output path is: >> hdfs://localhost:9000/user/hduser/output/shortestpaths >> 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Running Client >> 14/11/18 05:36:08 INFO client.RMProxy: Connecting to ResourceManager at >> localhost/127.0.0.1:8050 >> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Got node report from ASM for, >> nodeId=3Droosevelt:60486, nodeAddress roosevelt:8042, nodeRackName >> /default-rack, nodeNumContainers 0 >> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Defaulting per-task heap size >> to 1024MB. >> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Obtained new Application ID: >> application_1416310496070_0001 >> 14/11/18 05:36:09 INFO Configuration.deprecation: mapred.job.id is >> deprecated. Instead, use mapreduce.job.id >> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Set the environment for the >> application master >> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Environment for AM >> :{CLASSPATH=3D${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/h= >> adoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_H= >> OME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_Y= >> ARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HAD= >> OOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/m= >> apreduce/lib/*} >> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... >> 14/11/18 05:36:12 INFO yarn.YarnUtils: Registered file in LocalResources :: >> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= >> 496070_0001/giraph-conf.xml >> 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: LIB JARS >> :giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >> 14/11/18 05:36:12 INFO yarn.YarnUtils: Class path name . >> 14/11/18 05:36:12 INFO yarn.YarnUtils: base path checking . >> 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: Made local resource for >> :/usr/local/hadoop/share/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-ja= >> r-with-dependencies.jar >> to >> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= >> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= >> encies.jar >> 14/11/18 05:36:16 INFO yarn.YarnUtils: Registered file in LocalResources :: >> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= >> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= >> encies.jar >> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext >> for GiraphApplicationMaster launch container is populated. >> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Submitting application to ASM >> 14/11/18 05:36:16 INFO impl.YarnClientImpl: Submitted application >> application_1416310496070_0001 >> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Got new appId after >> submission :application_1416310496070_0001 >> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: GiraphApplicationMaster >> container request was submitted to ResourceManager for job: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation >> 14/11/18 05:36:17 INFO yarn.GiraphYarnClient: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 1.01 >> secs >> 14/11/18 05:36:17 INFO yarn.GiraphYarnClient: >> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 >> 14/11/18 05:36:22 INFO yarn.GiraphYarnClient: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.86 >> secs >> 14/11/18 05:36:22 INFO yarn.GiraphYarnClient: >> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 >> 14/11/18 05:36:26 INFO yarn.GiraphYarnClient: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 10.02 >> secs >> 14/11/18 05:36:26 INFO yarn.GiraphYarnClient: >> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 >> 14/11/18 05:36:30 INFO yarn.GiraphYarnClient: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 14.39 >> secs >> 14/11/18 05:36:30 INFO yarn.GiraphYarnClient: >> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 >> 14/11/18 05:36:34 INFO yarn.GiraphYarnClient: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 18.57 >> secs >> 14/11/18 05:36:34 INFO yarn.GiraphYarnClient: >> appattempt_1416310496070_0001_000001, State: RUNNING, Containers used: 2 >> >> gam_stderr.log >> >> LF4J: Class path contains multiple SLF4J bindings. >> SLF4J: Found binding in >> [jar:file:/app/hadoop/tmp/nm-local-dir/usercache/hduser/appcache/applicatio= >> n_1416310496070_0001/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop= >> -2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: Found binding in >> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar= >> !/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explana= >> tion. >> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >> >> gam_stdout.log >> >> 2014-11-18 05:36:28,143 INFO [main] yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:main(421)) - Starting GitaphAM >> 2014-11-18 05:36:30,404 WARN [main] util.NativeCodeLoader >> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop >> library for your platform... using builtin-java classes where >> applicable >> 2014-11-18 05:36:32,649 INFO [main] yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:<init>(168)) - GiraphAM for ContainerId >> container_1416310496070_0001_01_000001 ApplicationAttemptId >> appattempt_1416310496070_0001_000001 >> 2014-11-18 05:36:32,852 INFO [main] client.RMProxy >> (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at >> localhost/127.0.0.1:8030 >> 2014-11-18 05:36:32,888 INFO [main] impl.NMClientAsyncImpl >> (NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread >> pool size is 500 >> 2014-11-18 05:36:32,889 INFO [main] >> impl.ContainerManagementProtocolProxy >> (ContainerManagementProtocolProxy.java:<init>(78)) - >> yarn.client.max-nodemanagers-proxies : 500 >> 2014-11-18 05:36:33,589 INFO [main] yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested >> container ask: Capability[<memory:1024, vCores:0>]Priority[10] >> 2014-11-18 05:36:33,617 INFO [main] yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested >> container ask: Capability[<memory:1024, vCores:0>]Priority[10] >> 2014-11-18 05:36:33,617 INFO [main] yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:run(185)) - Wait to finish .. >> 2014-11-18 05:36:35,674 INFO [AMRM Heartbeater thread] >> impl.AMRMClientImpl (AMRMClientImpl.java:populateNMTokens(299)) - >> Received new token for : roosevelt:60486 >> 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread] >> yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got >> response from RM for container ask, allocatedCnt=3D1 >> 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread] >> yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total >> allocated # of container so far : 1 allocated out of 2 required. >> 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread] >> yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - >> Launching command on a new container., >> containerId=3Dcontainer_1416310496070_0001_01_000002, >> containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042, >> containerResourceMemory=3D1024 >> 2014-11-18 05:36:35,688 INFO [pool-4-thread-1] >> yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - >> Setting up container launch container for >> containerid=3Dcontainer_1416310496070_0001_01_000002 >> 2014-11-18 05:36:35,720 INFO [pool-4-thread-1] >> yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - >> Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH} >> org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 2 1 >> 1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log >> 2014-11-18 05:36:35,721 INFO [pool-4-thread-1] >> yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - >> Setting username in ContainerLaunchContext to: hduser >> 2014-11-18 05:36:36,789 INFO [AMRM Callback Handler Thread] >> yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got >> response from RM for container ask, allocatedCnt=3D1 >> 2014-11-18 05:36:36,789 INFO [AMRM Callback Handler Thread] >> yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total >> allocated # of container so far : 2 allocated out of 2 required. >> 2014-11-18 05:36:36,790 INFO [AMRM Callback Handler Thread] >> yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - >> Launching command on a new container., >> containerId=3Dcontainer_1416310496070_0001_01_000003, >> containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042, >> containerResourceMemory=3D1024 >> 2014-11-18 05:36:37,227 INFO [pool-4-thread-2] >> yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - >> Setting up container launch container for >> containerid=3Dcontainer_1416310496070_0001_01_000003 >> 2014-11-18 05:36:37,227 INFO [pool-4-thread-2] >> yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - >> Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH} >> org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 3 1 >> 1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log >> 2014-11-18 05:36:37,227 INFO [pool-4-thread-2] >> yarn.GiraphApplicationMaster >> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - >> Setting username in ContainerLaunchContext to: hduser >> 2014-11-18 05:36:37,463 INFO [pool-4-thread-1] yarn.YarnUtils >> (YarnUtils.java:addFsResourcesToMap(72)) - Adding >> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >> to LocalResources for export.to >> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= >> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= >> encies.jar >> 2014-11-18 05:36:37,678 INFO [pool-4-thread-1] yarn.YarnUtils >> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in >> LocalResources :: >> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= >> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= >> encies.jar >> 2014-11-18 05:36:37,678 WARN [pool-4-thread-1] yarn.YarnUtils >> (YarnUtils.java:addFsResourcesToMap(81)) - Job jars (-yj option) >> didn't include giraph-core. >> 2014-11-18 05:36:37,680 INFO [pool-4-thread-1] yarn.YarnUtils >> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in >> LocalResources :: >> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= >> 496070_0001/giraph-conf.xml >> 2014-11-18 05:36:37,694 INFO >> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] >> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing >> Event EventType: START_CONTAINER for Container >> container_1416310496070_0001_01_000002 >> 2014-11-18 05:36:37,699 INFO >> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1] >> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing >> Event EventType: START_CONTAINER for Container >> container_1416310496070_0001_01_000003 >> 2014-11-18 05:36:37,698 INFO >> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] >> impl.ContainerManagementProtocolProxy >> (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy >> : roosevelt:60486 >> 2014-11-18 05:36:37,761 INFO >> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2] >> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing >> Event EventType: QUERY_CONTAINER for Container >> container_1416310496070_0001_01_000003 >> 2014-11-18 05:36:37,771 INFO >> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #3] >> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing >> Event EventType: QUERY_CONTAINER for Container >> container_1416310496070_0001_01_000002 >> >> >> >> On Thu, Nov 20, 2014 at 1:32 PM, Alessandro Negro <[email protected]> >> wrote: >> >>> Hi Das, >>> I mean as user you run Hadoop yarn resource manager. >>> >>> Alessandro >>> >>> Il giorno 20/nov/2014, alle ore 19:22, D Adams <[email protected]> ha >>> scritto: >>> >>> Alessandro, >>> I'm not sure what you mean, should I create a new user on my system >>> named 'yarn'? I'm new to both hadoop and giraph, so I'm not sure. >>> >>> Tripti, >>> I'll get those logs as soon as I can. >>> >>> Thank you both, >>> >>> V/r >>> Das >>> On Nov 20, 2014 3:13 AM, "Alessandro Negro" <[email protected]> wrote: >>> >>>> Hi Tripti, >>>> I agree that a more detailed error log could be useful. >>>> >>>> Thanks, >>>> Alessandro >>>> >>>> >>>> >>>> >>>> Il giorno 19/nov/2014, alle ore 15:54, Tripti Singh <[email protected]= >> m> >>>> ha scritto: >>>> >>>>> --_000_D092AB9C1CB42triptiyahooinccom_ >>>>> Content-Type: text/plain; charset=3D"Windows-1252" >>>>> Content-Transfer-Encoding: quoted-printable >>>>> >>>>> Hi Alessandro, >>>>> >>>>> I guess u r talking about the documentation on Giraph webpage which >>>> mention=3D >>>>> s addition of this new option which is more or less mandatory for yarn >>>> base=3D >>>>> d profiles. >>>>> When I first ran the Giraph on yarn without the proper =3D96yj option, >>>> there =3D >>>>> was no way I could figure out from the Application/Container logs that >>>> this=3D >>>>> was the issue. >>>>> I think, it=3D92ll be a good idea to have this message in the logs for >>>> easy d=3D >>>>> ebugging. >>>>> >>>>> Thanks, >>>>> Tripti. >>>>> >>>>> >>>>> From: Alessandro Negro <[email protected]<mailto:[email protected] >>>>>> >>>>> Reply-To: "[email protected]<mailto:[email protected]>" >>>> <user@gir=3D >>>>> aph.apache.org<mailto:[email protected]>> >>>>> Date: Tuesday, November 18, 2014 at 4:31 PM >>>>> To: "[email protected]<mailto:[email protected]>" < >>>> [email protected]=3D >>>>> ache.org<mailto:[email protected]>> >>>>> Subject: Re: Help with Giraph on Yarn >>>>> >>>>> Hi Eli, >>>>> no I think that the help message is clear enough, generally when I rea= >> d >>>> =3D93=3D >>>>> jar=3D94 I always mean absolute path for jar, but this is a my persona= >> l >>>> misun=3D >>>>> derstanding. >>>>> >>>>> Thanks, >>>>> Alessandro >>>>> >>>>> >>>>> Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman < >>>> [email protected]=3D >>>>> m<mailto:[email protected]>> ha scritto: >>>>> >>>>> Sounds like you got it figured out since last time I checked this list= >> , >>>> sor=3D >>>>> ry it was a pain. Feel free to drop a JIRA ticket if we can make help >>>> messa=3D >>>>> ge etc. for -yj option more clear, there's lots to improve there. >>>>> >>>>> On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro <[email protected] >>>> <mailt=3D >>>>> o:[email protected]>> wrote: >>>>> Hi Tripti, >>>>> finally I was able to run the test with success. It was an issue of >>>> permiss=3D >>>>> ion since I was running as ale not as yarn. >>>>> Let me say that now I=3D92m able to run graph examples on Yarn 2.5.1. >>>> This is=3D >>>>> the final result: >>>>> >>>>> 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: >>>> org.apache.=3D >>>>> giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total >>>> running ti=3D >>>>> me: 0 minutes, 21 seconds. >>>>> >>>>> >>>>> Many thanks for your support, >>>>> Alessandro >>>>> >>>>> Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh < >>>> [email protected]<m=3D >>>>> ailto:[email protected]>> ha scritto: >>>>> >>>>> I don't know if u have access to this node. But if u do, u could check >>>> if t=3D >>>>> he file is indeed there and u have access to it. >>>>> >>>>> Sent from my iPhone >>>>> >>>>> On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" <[email protected] >>>> <mailto:=3D >>>>> [email protected]>> wrote: >>>>> >>>>> You are right it works, but now I receive the following error: >>>>> >>>>> SLF4J: Class path contains multiple SLF4J bindings. >>>>> SLF4J: Found binding in >>>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D >>>>> >>>> rcache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-e= >> xam=3D >>>>> >>>> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4= >> j/i=3D >>>>> mpl/StaticLoggerBinder.class] >>>>> SLF4J: Found binding in >>>> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D >>>>> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class= >> ] >>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >>>> explana=3D >>>>> tion. >>>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >>>>> 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info fro= >> m >>>> SCD=3D >>>>> ynamicStore >>>>> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: >>>> Coul=3D >>>>> d not configure the containerlaunch context for GiraphYarnTasks. >>>>> at >>>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= >> =3D >>>>> ApplicationMaster.java:391) >>>>> at >>>> org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicat= >> =3D >>>>> ionMaster.java:78) >>>>> at >>>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.b= >> =3D >>>>> uildContainerLaunchContext(GiraphApplicationMaster.java:522) >>>>> at >>>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.r= >> =3D >>>>> un(GiraphApplicationMaster.java:479) >>>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= >> =3D >>>>> a:1145) >>>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= >> =3D >>>>> va:615) >>>>> at java.lang.Thread.run(Thread.java:744) >>>>> Caused by: java.io.FileNotFoundException: File does not exist: >>>> hdfs://hadoo=3D >>>>> >>>> p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_= >> 000=3D >>>>> >>>> 9/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.= >> jar >>>>> at >>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= >> =3D >>>>> stem.java:1072) >>>>> at >>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= >> =3D >>>>> stem.java:1064) >>>>> at >>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv= >> =3D >>>>> er.java:81) >>>>> at >>>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi= >> =3D >>>>> leSystem.java:1064) >>>>> at >>>> org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153= >> =3D >>>>> ) >>>>> at >>>> org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) >>>>> at >>>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= >> =3D >>>>> ApplicationMaster.java:387) >>>>> ... 6 more >>>>> >>>>> >>>>> That justify the other error I receive in the task: >>>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas= >> k >>>>> >>>>> Thanks, >>>>> >>>>> Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh < >>>> [email protected]<m=3D >>>>> ailto:[email protected]>> ha scritto: >>>>> >>>>> Why r u adding two jars? Example jar ideally contains core library so >>>> every=3D >>>>> thing should be available with just one example jar included >>>>> >>>>> Sent from my iPhone >>>>> >>>>> On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <[email protected] >>>> <mailto:=3D >>>>> [email protected]>> wrote: >>>>> >>>>> Hi, >>>>> now it seems better, I need to add: >>>>> >>>>> >>>> giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-= >> exa=3D >>>>> mples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>>>> >>>>> >>>>> Now it seems that after a lot of cycle it fail with this error: >>>>> >>>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas= >> k >>>>> >>>>> But in this case the error appear in task-3-stderr.log not in >>>> gam-stderr.l=3D >>>>> og where there is the following error: >>>>> >>>>> LF4J: Class path contains multiple SLF4J bindings. >>>>> SLF4J: Found binding in >>>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D >>>>> >>>> rcache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1= >> .1.=3D >>>>> >>>> 0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/St= >> ati=3D >>>>> cLoggerBinder.class] >>>>> SLF4J: Found binding in >>>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D >>>>> >>>> rcache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-e= >> xam=3D >>>>> >>>> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4= >> j/i=3D >>>>> mpl/StaticLoggerBinder.class] >>>>> SLF4J: Found binding in >>>> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D >>>>> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class= >> ] >>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >
