Hi Kyle,

Thanks for your question and welcome to Giraph! It looks like you couldn't get enough resources for the test to run on your hadoop instance. In this example, you are asking for 30 workers. You will need to be able to get 30 + 1 (master) = 31 map tasks to start the test. If Giraph can't get all 31 map tasks within a period of time, it will fail. Are you submitting this to an actual Hadoop cluster with at least 31 available map tasks?


Avery

On 9/7/11 2:13 PM, Kyle Teague wrote:
I am trying to run the following command in pseudo-distributed mode
from the Getting Started example page: hadoop jar
giraph-0.70-jar-with-dependencies.jar
org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000
-w 30

Here is the task log output:

2011-09-07 15:41:34,311 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2011-09-07 15:41:34,529 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
already exists!
2011-09-07 15:41:34,641 WARN org.apache.giraph.bsp.BspOutputFormat:
getOutputCommitter: Returning ImmutableOutputCommiter (does nothing).
2011-09-07 15:41:34,688 INFO org.apache.giraph.graph.GraphMapper:
setup: jar file @
/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar,
using 
/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar
2011-09-07 15:41:34,694 INFO org.apache.giraph.zk.ZooKeeperManager:
createCandidateStamp: Made the directory
_bsp/_defaultZkManagerDir/job_201109071501_0003
2011-09-07 15:41:34,695 INFO org.apache.giraph.zk.ZooKeeperManager:
createCandidateStamp: Creating my filestamp
_bsp/_defaultZkManagerDir/job_201109071501_0003/_task/new-host-3.home
0
2011-09-07 15:41:34,710 INFO org.apache.giraph.zk.ZooKeeperManager:
getZooKeeperServerList: Got [new-host-3.home] 1 hosts from 1
candidates when 1 required (polling period is 3000) on attempt 0
2011-09-07 15:41:34,711 INFO org.apache.giraph.zk.ZooKeeperManager:
createZooKeeperServerList: Creating the final ZooKeeper file
'_bsp/_defaultZkManagerDir/job_201109071501_0003/zkServerList_new-host-3.home
0 '
2011-09-07 15:41:34,717 INFO org.apache.giraph.zk.ZooKeeperManager:
getZooKeeperServerList: For task 0, got file
'zkServerList_new-host-3.home 0 ' (polling period is 3000)
2011-09-07 15:41:34,718 INFO org.apache.giraph.zk.ZooKeeperManager:
getZooKeeperServerList: Found [new-host-3.home, 0] 2 hosts in filename
'zkServerList_new-host-3.home 0'
2011-09-07 15:41:34,720 INFO org.apache.giraph.zk.ZooKeeperManager:
onlineZooKeeperServers: Trying to delete old directory
/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper
2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager:
generateZooKeeperConfigFile: Creating file
/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg
in 
/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper
with base port 22181
2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager:
generateZooKeeperConfigFile: Make directory of _bspZooKeeper = true
2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager:
generateZooKeeperConfigFile: Delete of zoo.cfg = false
2011-09-07 15:41:34,726 INFO org.apache.giraph.zk.ZooKeeperManager:
onlineZooKeeperServers: Attempting to start ZooKeeper server with
command 
[/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java,
-Xmx256m, -XX:ParallelGCThreads=4, -XX:+UseConcMarkSweepGC,
-XX:CMSInitiatingOccupancyFraction=70, -XX:MaxGCPauseMillis=100, -cp,
/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar,
org.apache.zookeeper.server.quorum.QuorumPeerMain,
/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg]
in directory 
/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper
2011-09-07 15:41:34,748 INFO org.apache.giraph.zk.ZooKeeperManager:
onlineZooKeeperServers: Connect attempt 0 of 10 max trying to connect
to new-host-3.home:22181 with poll msecs = 3000
2011-09-07 15:41:34,775 WARN org.apache.giraph.zk.ZooKeeperManager:
onlineZooKeeperServers: Got ConnectException
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
        at java.net.Socket.connect(Socket.java:529)
        at 
org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:611)
        at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:419)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:253)
2011-09-07 15:41:37,776 INFO org.apache.giraph.zk.ZooKeeperManager:
onlineZooKeeperServers: Connect attempt 1 of 10 max trying to connect
to new-host-3.home:22181 with poll msecs = 3000
2011-09-07 15:41:37,777 INFO org.apache.giraph.zk.ZooKeeperManager:
onlineZooKeeperServers: Connected to
new-host-3.home/192.168.1.6:22181!
2011-09-07 15:41:37,777 INFO org.apache.giraph.zk.ZooKeeperManager:
onlineZooKeeperServers: Creating my filestamp
_bsp/_defaultZkManagerDir/job_201109071501_0003/_zkServer/new-host-3.home
0
2011-09-07 15:41:37,782 INFO org.apache.giraph.graph.GraphMapper:
setup: Starting up BspServiceMaster (master thread)...
2011-09-07 15:41:37,791 INFO org.apache.giraph.graph.BspService:
BspService: Connecting to ZooKeeper with job job_201109071501_0003, 0
on new-host-3.home:22181
2011-09-07 15:41:37,797 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
GMT
2011-09-07 15:41:37,797 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=new-host-3.home
2011-09-07 15:41:37,797 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_26
2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Apple Inc.
2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/classes:/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars:/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work:/Users/kyle/hadoop/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Home//lib/tools.jar:/Users/kyle/hadoop/bin/..:/Users/kyle/hadoop/bin/../hadoop-core-0.20.203.0.jar:/Users/kyle/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/Users/kyle/hadoop/bin/../lib/aspectjtools-1.6.5.jar:/Users/kyle/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/Users/kyle/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:/Users/kyle/hadoop/bin/../lib/commons-cli-1.2.jar:/Users/kyle/hadoop/bin/../lib/commons-codec-1.4.jar:/Users/kyle/hadoop/bin/../lib/commons-collections-3.2.1.jar:/Users/kyle/hadoop/bin/../lib/commons-configuration-1.6.jar:/Users/kyle/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/Users/kyle/hadoop/bin/../lib/commons-digester-1.8.jar:/Users/kyle/hadoop/bin/../lib/commons-el-1.0.jar:/Users/kyle/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:/Users/kyle/hadoop/bin/../lib/commons-lang-2.4.jar:/Users/kyle/hadoop/bin/../lib/commons-logging-1.1.1.jar:/Users/kyle/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/Users/kyle/hadoop/bin/../lib/commons-math-2.1.jar:/Users/kyle/hadoop/bin/../lib/commons-net-1.4.1.jar:/Users/kyle/hadoop/bin/../lib/core-3.1.1.jar:/Users/kyle/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/Users/kyle/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:/Users/kyle/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/Users/kyle/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:/Users/kyle/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/Users/kyle/hadoop/bin/../lib/jets3t-0.6.1.jar:/Users/kyle/hadoop/bin/../lib/jetty-6.1.26.jar:/Users/kyle/hadoop/bin/../lib/jetty-util-6.1.26.jar:/Users/kyle/hadoop/bin/../lib/jsch-0.1.42.jar:/Users/kyle/hadoop/bin/../lib/junit-4.5.jar:/Users/kyle/hadoop/bin/../lib/kfs-0.2.2.jar:/Users/kyle/hadoop/bin/../lib/log4j-1.2.15.jar:/Users/kyle/hadoop/bin/../lib/mockito-all-1.8.5.jar:/Users/kyle/hadoop/bin/../lib/oro-2.0.8.jar:/Users/kyle/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/Users/kyle/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/Users/kyle/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/Users/kyle/hadoop/bin/../lib/xmlenc-0.52.jar:/Users/kyle/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/Users/kyle/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar
2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/Users/kyle/hadoop/bin/../lib/native/Mac_OS_X-x86_64-64:/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work
2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work/tmp
2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Mac OS X
2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=x86_64
2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=10.6.8
2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=kyle
2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/homes/
2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/private/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work
2011-09-07 15:41:37,799 INFO org.apache.zookeeper.ZooKeeper:
Initiating client connection, connectString=new-host-3.home:22181
sessionTimeout=60000
watcher=org.apache.giraph.graph.BspServiceMaster@769aba32
2011-09-07 15:41:37,810 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server new-host-3.home/192.168.1.6:22181
2011-09-07 15:41:37,811 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to new-host-3.home/192.168.1.6:22181,
initiating session
2011-09-07 15:41:37,855 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server new-host-3.home/192.168.1.6:22181,
sessionid = 0x1324568e60f0000, negotiated timeout = 60000
2011-09-07 15:41:37,856 INFO org.apache.giraph.graph.BspService:
process: Asynchronous connection complete.
2011-09-07 15:41:37,857 INFO org.apache.giraph.graph.GraphMapper: map:
No need to do anything when not a worker
2011-09-07 15:41:37,857 INFO org.apache.giraph.graph.GraphMapper:
cleanup: Starting for MASTER_ZOOKEEPER_ONLY
2011-09-07 15:41:37,907 INFO org.apache.giraph.graph.BspServiceMaster:
becomeMaster: First child is
'/_hadoopBsp/job_201109071501_0003/_masterElectionDir/new-host-3.home_00000000000'
and my bid is 
'/_hadoopBsp/job_201109071501_0003/_masterElectionDir/new-host-3.home_00000000000'
2011-09-07 15:41:37,907 INFO org.apache.giraph.graph.BspServiceMaster:
becomeMaster: I am now the master!
2011-09-07 15:41:37,918 INFO org.apache.giraph.graph.BspService:
process: applicationAttemptChanged signaled
2011-09-07 15:41:37,926 WARN org.apache.giraph.graph.BspService:
process: Unknown and unprocessed event
(path=/_hadoopBsp/job_201109071501_0003/_applicationAttemptsDir/0/_superstepDir,
type=NodeChildrenChanged, state=SyncConnected)
2011-09-07 15:42:10,510 INFO org.apache.giraph.graph.BspServiceMaster:
checkWorkers: Only found 1 responses of 30 needed to start superstep
-1.  Sleeping for 30000 msecs and used 0 of 10 attempts.
2011-09-07 15:42:40,514 INFO org.apache.giraph.graph.BspServiceMaster:
checkWorkers: Only found 1 responses of 30 needed to start superstep
-1.  Sleeping for 30000 msecs and used 1 of 10 attempts.
2011-09-07 15:43:10,519 INFO org.apache.giraph.graph.BspServiceMaster:
checkWorkers: Only found 1 responses of 30 needed to start superstep
-1.  Sleeping for 30000 msecs and used 2 of 10 attempts.
2011-09-07 15:43:40,523 INFO org.apache.giraph.graph.BspServiceMaster:
checkWorkers: Only found 1 responses of 30 needed to start superstep
-1.  Sleeping for 30000 msecs and used 3 of 10 attempts.
2011-09-07 15:44:10,527 INFO org.apache.giraph.graph.BspServiceMaster:
checkWorkers: Only found 1 responses of 30 needed to start superstep
-1.  Sleeping for 30000 msecs and used 4 of 10 attempts.
2011-09-07 15:44:40,533 INFO org.apache.giraph.graph.BspServiceMaster:
checkWorkers: Only found 1 responses of 30 needed to start superstep
-1.  Sleeping for 30000 msecs and used 5 of 10 attempts.
2011-09-07 15:45:10,537 INFO org.apache.giraph.graph.BspServiceMaster:
checkWorkers: Only found 1 responses of 30 needed to start superstep
-1.  Sleeping for 30000 msecs and used 6 of 10 attempts.
2011-09-07 15:45:40,541 INFO org.apache.giraph.graph.BspServiceMaster:
checkWorkers: Only found 1 responses of 30 needed to start superstep
-1.  Sleeping for 30000 msecs and used 7 of 10 attempts.
2011-09-07 15:46:10,545 INFO org.apache.giraph.graph.BspServiceMaster:
checkWorkers: Only found 1 responses of 30 needed to start superstep
-1.  Sleeping for 30000 msecs and used 8 of 10 attempts.
2011-09-07 15:46:40,550 INFO org.apache.giraph.graph.BspServiceMaster:
checkWorkers: Only found 1 responses of 30 needed to start superstep
-1.  Sleeping for 30000 msecs and used 9 of 10 attempts.
2011-09-07 15:46:40,550 WARN org.apache.giraph.graph.BspServiceMaster:
checkWorkers: Did not receive enough processes in time (only 1 of 30
required)
2011-09-07 15:46:40,552 INFO org.apache.giraph.graph.BspServiceMaster:
setJobState: 
{"_stateKey":"FAILED","_applicationAttemptKey":-1,"_superstepKey":-1}
on superstep -1
2011-09-07 15:46:41,344 FATAL
org.apache.giraph.graph.BspServiceMaster: failJob: Killing job
job_201109071501_0003
2011-09-07 15:46:41,378 ERROR org.apache.giraph.graph.MasterThread:
masterThread: Master algorithm failed:
java.lang.NullPointerException
        at 
org.apache.giraph.graph.BspServiceMaster.createInputSplits(BspServiceMaster.java:486)
        at org.apache.giraph.graph.MasterThread.run(MasterThread.java:94)
2011-09-07 15:46:41,379 FATAL org.apache.giraph.graph.GraphMapper:
uncaughtException: OverrideExceptionHandler on thread
org.apache.giraph.graph.MasterThread, msg =
java.lang.NullPointerException, exiting...
java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.giraph.graph.MasterThread.run(MasterThread.java:177)
Caused by: java.lang.NullPointerException
        at 
org.apache.giraph.graph.BspServiceMaster.createInputSplits(BspServiceMaster.java:486)
        at org.apache.giraph.graph.MasterThread.run(MasterThread.java:94)
2011-09-07 15:46:41,379 WARN org.apache.giraph.zk.ZooKeeperManager:
onlineZooKeeperServers: Forced a shutdown hook kill of the ZooKeeper
process.


Reply via email to