I am sure the machines can communicate to each other and the ports are not blocked. I can run word count hadoop job without any problem on these machines. My hadoop version is 0.20.203.0.
Thanks, Inci On Dec 1, 2011, at 3:57 PM, Avery Ching wrote: > Thanks for the logs. I see a lot of issues like the following: > > 2011-12-01 00:04:46,241 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: rainbow-01/192.168.100.1:30004. Already tried 0 time(s). > 2011-12-01 00:04:47,243 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: rainbow-01/192.168.100.1:30004. Already tried 1 time(s). > 2011-12-01 00:04:48,245 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: rainbow-01/192.168.100.1:30004. Already tried 2 time(s). > 2011-12-01 00:04:49,247 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: rainbow-01/192.168.100.1:30004. Already tried 3 time(s). > 2011-12-01 00:04:50,249 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: rainbow-01/192.168.100.1:30004. Already tried 4 time(s). > 2011-12-01 00:04:51,251 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: rainbow-01/192.168.100.1:30004. Already tried 5 time(s). > 2011-12-01 00:04:52,253 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: rainbow-01/192.168.100.1:30004. Already tried 6 time(s). > 2011-12-01 00:04:53,255 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: rainbow-01/192.168.100.1:30004. Already tried 7 time(s). > 2011-12-01 00:04:54,256 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: rainbow-01/192.168.100.1:30004. Already tried 8 time(s). > 2011-12-01 00:04:55,258 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: rainbow-01/192.168.100.1:30004. Already tried 9 time(s). > 2011-12-01 00:04:55,261 WARN org.apache.giraph.comm.BasicRPCCommunications: > connectAllRPCProxys: Failed on attempt 0 of 5 to connect to > (id=0,cur=Worker(hostname=rainbow-01, MRpartition=4, > port=30004),prev=null,ckpt_file=null) > java.net.ConnectException: Call to rainbow-01/192.168.100.1:30004 failed on > connection exception: java.net.ConnectException: Connection refused > > Are you sure that your machines can communicate to each other? Are the ports > 30000 and up blocked? And you're right, you should have only had 6 tasks. > What version of Hadoop is this on? > > Avery > > On 12/1/11 2:43 PM, Inci Cetindil wrote: >> >> Hi Avery, >> >> I attached the logs for the first attemps. The weird thing is even if I >> specified the number of workers as 5, I had 8 mapper tasks. You can see the >> logs for tasks 6 and 7 failed immediately. Do you have any explanation for >> this behavior? Normally I should have 6 tasks, right? >> >> Thanks, >> Inci >> >> >> >> >> On Dec 1, 2011, at 11:00 AM, Avery Ching wrote: >> >>> Hi Inci, >>> >>> I am not sure what's wrong. I ran the exact same command on a freshly >>> checked version of Graph without any trouble. Here's my output: >>> >>> hadoop jar target/giraph-0.70-jar-with-dependencies.jar >>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 500 -w 5 >>> Using org.apache.giraph.benchmark.PageRankBenchmark$PageRankVertex >>> 11/12/01 10:58:05 WARN bsp.BspOutputFormat: checkOutputSpecs: >>> ImmutableOutputCommiter will not check anything >>> 11/12/01 10:58:05 INFO mapred.JobClient: Running job: job_201112011054_0003 >>> 11/12/01 10:58:06 INFO mapred.JobClient: map 0% reduce 0% >>> 11/12/01 10:58:23 INFO mapred.JobClient: map 16% reduce 0% >>> 11/12/01 10:58:35 INFO mapred.JobClient: map 100% reduce 0% >>> 11/12/01 10:58:40 INFO mapred.JobClient: Job complete: job_201112011054_0003 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Counters: 31 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Job Counters >>> 11/12/01 10:58:40 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=77566 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Total time spent by all >>> reduces waiting after reserving slots (ms)=0 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Total time spent by all maps >>> waiting after reserving slots (ms)=0 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Launched map tasks=6 >>> 11/12/01 10:58:40 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Giraph Timers >>> 11/12/01 10:58:40 INFO mapred.JobClient: Total (milliseconds)=13468 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Superstep 3 (milliseconds)=41 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Setup (milliseconds)=11691 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Shutdown (milliseconds)=73 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Vertex input superstep >>> (milliseconds)=369 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Superstep 0 (milliseconds)=674 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Superstep 2 (milliseconds)=519 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Superstep 1 (milliseconds)=96 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Giraph Stats >>> 11/12/01 10:58:40 INFO mapred.JobClient: Aggregate edges=500 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Superstep=4 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Last checkpointed superstep=2 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Current workers=5 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Current master task partition=0 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Sent messages=0 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Aggregate finished vertices=500 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Aggregate vertices=500 >>> 11/12/01 10:58:40 INFO mapred.JobClient: File Output Format Counters >>> 11/12/01 10:58:40 INFO mapred.JobClient: Bytes Written=0 >>> 11/12/01 10:58:40 INFO mapred.JobClient: FileSystemCounters >>> 11/12/01 10:58:40 INFO mapred.JobClient: FILE_BYTES_READ=590 >>> 11/12/01 10:58:40 INFO mapred.JobClient: HDFS_BYTES_READ=264 >>> 11/12/01 10:58:40 INFO mapred.JobClient: FILE_BYTES_WRITTEN=129240 >>> 11/12/01 10:58:40 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=55080 >>> 11/12/01 10:58:40 INFO mapred.JobClient: File Input Format Counters >>> 11/12/01 10:58:40 INFO mapred.JobClient: Bytes Read=0 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Map-Reduce Framework >>> 11/12/01 10:58:40 INFO mapred.JobClient: Map input records=6 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Spilled Records=0 >>> 11/12/01 10:58:40 INFO mapred.JobClient: Map output records=0 >>> 11/12/01 10:58:40 INFO mapred.JobClient: SPLIT_RAW_BYTES=264 >>> >>> >>> Would it be possible to send me the logs from the first attempts for every >>> map task? >>> >>> i.e. from >>> Task attempt_201111302343_0002_m_000000_0 >>> Task attempt_201111302343_0002_m_000001_0 >>> Task attempt_201111302343_0002_m_000002_0 >>> Task attempt_201111302343_0002_m_000003_0 >>> Task attempt_201111302343_0002_m_000004_0 >>> Task attempt_201111302343_0002_m_000005_0 >>> >>> I think that could help us find the issue. >>> >>> Thanks, >>> >>> Avery >>> >>> On 12/1/11 1:17 AM, Inci Cetindil wrote: >>>> Hi, >>>> >>>> I'm running PageRank benchmark example on a cluster with 1 master + 5 >>>> slave nodes. I have tried it with a large number of vertices; when I >>>> failed I decided to make it run with 500 vertices and 5 workers first. >>>> However, it doesn't work even for 500 vertices. >>>> I am using the latest version of Giraph from the trunk and running the >>>> following command: >>>> >>>> hadoop jar giraph-0.70-jar-with-dependencies.jar >>>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 500 -w 5 >>>> >>>> I attached the error message that I am receiving. Please let me know if I >>>> am missing something. >>>> >>>> Best regards, >>>> Inci >>>> >>>> >>>> >>>> >>>> >
