OK, got by this problem. It turns out, I built Giraph using the hadoop_2 profile, and not hadoop_yarn. When I switched to using mvn -Phadoop_yarn, the problem I was experiencing changed... I now see log messages showing Girpah connecting to the Yarn ResourceManager on port 8032, which is good. Unfortunately I now get a *different* error related to the heap size, but I'll post that in a separate message.
Phil This message optimized for indexing by NSA PRISM On Mon, Mar 9, 2015 at 11:39 PM, Phillip Rhodes <[email protected]> wrote: > Folks: > > I'm trying to run the basic ShortestPaths example using Giraph 1.1 on > Hadoop 2.5.2. I'm running in actual cluster model (eg, not > psuedo-distributed) and I can run standard mapreduce jobs OK. But > when I try to run the Giraph example, it seems to hang unless I set > > -ca giraph.SplitMasterWorker=false > > and correspondingly set number of workers to 1. But this kinda > defeats the point of running on a cluster, no? OTOH, if I run > without disabling SplitMasterWorker, I get this error: > > When using LocalJobRunner, you cannot run in split master / worker mode > since there is only 1 task at a time! > > which suggests that Girpah is defaulting to local mode. One report I > read suggested fixing this by adding > > -ca mapred.job.tracker=10.0.0.12:5431 > > to the Girpah command line, but on Hadoop 2.5.2 with YARN, there is no > JobTracker on port 5431, if I understand correctly. Anyway, if I do > add that bit, the job **tries** to run, but seems to hang without ever > finishing. Here's the complete command line, and the job output > follows: > > > [prhodes@ip-10-0-0-12 conf]$ hadoop jar /home/prhodes/giraph/giraph- > examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with- > dependencies.jar org.apache.giraph.GiraphRunner > org.apache.giraph.examples.SimpleShortestPathsComputation -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat > -vip /user/prhodes/input/tiny_graph.txt -vof > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op > /user/prhodes/giraph_output/shortestpaths -w 3 -ca > mapred.job.tracker=10.0.0.12:5431 > > > > > 15/03/10 03:18:59 INFO utils.ConfigurationUtils: No edge input > format specified. Ensure your InputFormat does not require one. > 15/03/10 03:19:02 INFO server.NIOServerCnxnFactory: binding to > port 0.0.0.0/0.0.0.0:22181 > 15/03/10 03:19:02 INFO server.PrepRequestProcessor: > zookeeper.skipACL=="yes", ACL checks will be skipped > 15/03/10 03:19:05 INFO zk.ZooKeeperManager: > onlineZooKeeperServers: Connect attempt 1 of 10 max trying to connect > to ip-10-0-0-12.ec2.internal:22181 with poll msecs = 3000 > 15/03/10 03:19:05 INFO zk.ZooKeeperManager: > onlineZooKeeperServers: Connected to > ip-10-0-0-12.ec2.internal/10.0.0.12:22181! > 15/03/10 03:19:05 INFO zk.ZooKeeperManager: > onlineZooKeeperServers: Creating my filestamp > _bsp/_defaultZkManagerDir/job_local1346154675_0001/_zkServer/ip-10-0-0-12.ec2.internal > 0 > 15/03/10 03:19:05 INFO server.NIOServerCnxnFactory: Accepted > socket connection from /10.0.0.12:45182 > 15/03/10 03:19:05 INFO graph.GraphTaskManager: setup: Chosen to > run ZooKeeper... > 15/03/10 03:19:05 INFO graph.GraphTaskManager: setup: Starting up > BspServiceMaster (master thread)... > 15/03/10 03:19:05 INFO bsp.BspService: BspService: Path to create > to halt is /_hadoopBsp/job_local1346154675_0001/_haltComputation > 15/03/10 03:19:05 INFO bsp.BspService: BspService: Connecting to > ZooKeeper with job job_local1346154675_0001, 0 on > ip-10-0-0-12.ec2.internal:22181 > 15/03/10 03:19:05 INFO zookeeper.ClientCnxn: Opening socket > connection to server ip-10-0-0-12.ec2.internal/10.0.0.12:22181. Will > not attempt to authenticate using SASL (unknown error) > 15/03/10 03:19:05 INFO server.NIOServerCnxnFactory: Accepted > socket connection from /10.0.0.12:45183 > 15/03/10 03:19:05 INFO zookeeper.ClientCnxn: Socket connection > established to ip-10-0-0-12.ec2.internal/10.0.0.12:22181, initiating > session > 15/03/10 03:19:05 INFO server.ZooKeeperServer: Client attempting > to establish new session at /10.0.0.12:45183 > 15/03/10 03:19:05 INFO persistence.FileTxnLog: Creating new log file: > log.1 > 15/03/10 03:19:05 INFO server.ZooKeeperServer: Established session > 0x14c01b158f00000 with negotiated timeout 600000 for client > /10.0.0.12:45183 > 15/03/10 03:19:05 INFO zookeeper.ClientCnxn: Session establishment > complete on server ip-10-0-0-12.ec2.internal/10.0.0.12:22181, > sessionid = 0x14c01b158f00000, negotiated timeout = 600000 > 15/03/10 03:19:05 INFO bsp.BspService: process: Asynchronous > connection complete. > 15/03/10 03:19:05 INFO graph.GraphTaskManager: map: No need to do > anything when not a worker > 15/03/10 03:19:05 INFO graph.GraphTaskManager: cleanup: Starting > for MASTER_ZOOKEEPER_ONLY > 15/03/10 03:19:05 INFO server.PrepRequestProcessor: Got user-level > KeeperException when processing sessionid:0x14c01b158f00000 > type:create cxid:0x1 zxid:0x2 txntype:-1 reqpath:n/a Error > Path:/_hadoopBsp/job_local1346154675_0001/_masterElectionDir > Error:KeeperErrorCode = NoNode for > /_hadoopBsp/job_local1346154675_0001/_masterElectionDir > 15/03/10 03:19:05 INFO master.BspServiceMaster: becomeMaster: > First child is > '/_hadoopBsp/job_local1346154675_0001/_masterElectionDir/ip-10-0-0-12.ec2.internal_00000000000' > and my bid is > '/_hadoopBsp/job_local1346154675_0001/_masterElectionDir/ip-10-0-0-12.ec2.internal_00000000000' > 15/03/10 03:19:05 INFO netty.NettyServer: NettyServer: Using > execution group with 8 threads for requestFrameDecoder. > 15/03/10 03:19:05 INFO Configuration.deprecation: mapred.map.tasks > is deprecated. Instead, use mapreduce.job.maps > 15/03/10 03:19:05 INFO netty.NettyServer: start: Started server > communication server: ip-10-0-0-12.ec2.internal/10.0.0.12:30000 with > up to 16 threads on bind attempt 0 with sendBufferSize = 32768 > receiveBufferSize = 524288 > 15/03/10 03:19:05 INFO netty.NettyClient: NettyClient: Using > execution handler with 8 threads after request-encoder. > 15/03/10 03:19:05 INFO master.BspServiceMaster: becomeMaster: I am > now the master! > 15/03/10 03:19:05 INFO server.PrepRequestProcessor: Got user-level > KeeperException when processing sessionid:0x14c01b158f00000 > type:create cxid:0xe zxid:0x9 txntype:-1 reqpath:n/a Error > Path:/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0 > Error:KeeperErrorCode = NoNode for > /_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0 > 15/03/10 03:19:05 INFO bsp.BspService: process: > applicationAttemptChanged signaled > 15/03/10 03:19:05 INFO server.PrepRequestProcessor: Got user-level > KeeperException when processing sessionid:0x14c01b158f00000 > type:create cxid:0x16 zxid:0xc txntype:-1 reqpath:n/a Error > Path:/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1 > Error:KeeperErrorCode = NoNode for > /_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1 > 15/03/10 03:19:05 WARN bsp.BspService: process: Unknown and > unprocessed event > (path=/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir, > type=NodeChildrenChanged, state=SyncConnected) > 15/03/10 03:19:07 INFO mapred.LocalJobRunner: > MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed > to start superstep -1 > map > 15/03/10 03:19:07 INFO > job.HaltApplicationUtils$DefaultHaltInstructionsWriter: > writeHaltInstructions: To halt after next superstep execute: > 'bin/halt-application --zkServer ip-10-0-0-12.ec2.internal:22181 > --zkNode /_hadoopBsp/job_local1346154675_0001/_haltComputation' > 15/03/10 03:19:07 INFO mapreduce.Job: Running job: > job_local1346154675_0001 > 15/03/10 03:19:08 INFO mapreduce.Job: Job job_local1346154675_0001 > running in uber mode : false > 15/03/10 03:19:08 INFO mapreduce.Job: map 25% reduce 0% > 15/03/10 03:19:10 INFO mapred.LocalJobRunner: > MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed > to start superstep -1 > map > 15/03/10 03:19:19 INFO mapred.LocalJobRunner: > MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed > to start superstep -1 > map > 15/03/10 03:19:28 INFO mapred.LocalJobRunner: > MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed > to start superstep -1 > map > 15/03/10 03:19:35 INFO master.BspServiceMaster: checkWorkers: Only > found 0 responses of 3 needed to start superstep -1. Reporting every > 30000 msecs, 569976 more msecs left before giving up. > 15/03/10 03:19:35 INFO server.PrepRequestProcessor: Got user-level > KeeperException when processing sessionid:0x14c01b158f00000 > type:create cxid:0x22 zxid:0x10 txntype:-1 reqpath:n/a Error > Path:/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerHealthyDir > Error:KeeperErrorCode = NodeExists for > /_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerHealthyDir > 15/03/10 03:19:35 INFO server.PrepRequestProcessor: Got user-level > KeeperException when processing sessionid:0x14c01b158f00000 > type:create cxid:0x23 zxid:0x11 txntype:-1 reqpath:n/a Error > Path:/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerUnhealthyDir > Error:KeeperErrorCode = NodeExists for > /_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerUnhealthyDir > 15/03/10 03:19:40 INFO mapred.LocalJobRunner: > MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed > to start superstep -1 > map > > Any help is much appreciated! > > > Phil
