Funny how log outputs coming into this directory. There shouldn't be any logs. I create a filter for that later on.
Thanks for the testing Edward. 2011/5/18 Edward J. Yoon <[email protected]> > FYI, > > Below is the test result on 16 nodes (16 * 16 cores) > > ---- > root@Cnode1:/usr/local/src/hama-trunk# bin/hama jar > hama-0.3.0-examples.jar sssp Klewno /user/root/output /user/root/d > Single Source Shortest Path Example: > <Startvertex name> <optional: output path> <optional: path to own adjacency > list > Setting default start vertex to "Frankfurt"! > Setting start vertex to Klewno! > Using new output folder: /user/root/output > 11/05/18 16:44:23 INFO examples.ShortestPaths: Starting data > partitioning... > 11/05/18 16:44:23 WARN util.NativeCodeLoader: Unable to load native-hadoop > libra > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new compressor > 11/05/18 16:44:23 INFO compress.CodecPool: Got brand-new decompressor > 11/05/18 16:51:38 INFO examples.ShortestPaths: Finished! > 11/05/18 16:51:38 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir: > hdfs://hnode15:9000/tmp/hadoop-root/bsp/system/su > 11/05/18 16:51:38 INFO bsp.BSPJobClient: Running job: job_201105181515_0002 > 11/05/18 16:51:41 INFO bsp.BSPJobClient: Current supersteps number: 0 > 11/05/18 16:52:02 INFO bsp.BSPJobClient: Current supersteps number: 1 > 11/05/18 16:52:11 INFO bsp.BSPJobClient: Current supersteps number: 2 > 11/05/18 16:52:20 INFO bsp.BSPJobClient: Current supersteps number: 3 > 11/05/18 16:52:32 INFO bsp.BSPJobClient: Current supersteps number: 4 > 11/05/18 16:52:41 INFO bsp.BSPJobClient: Current supersteps number: 5 > 11/05/18 16:52:53 INFO bsp.BSPJobClient: Current supersteps number: 6 > 11/05/18 16:53:02 INFO bsp.BSPJobClient: Current supersteps number: 7 > 11/05/18 16:53:14 INFO bsp.BSPJobClient: Current supersteps number: 8 > 11/05/18 16:53:23 INFO bsp.BSPJobClient: Current supersteps number: 9 > 11/05/18 16:53:35 INFO bsp.BSPJobClient: Current supersteps number: 10 > 11/05/18 16:53:44 INFO bsp.BSPJobClient: Current supersteps number: 11 > 11/05/18 16:53:56 INFO bsp.BSPJobClient: Current supersteps number: 12 > 11/05/18 16:54:08 INFO bsp.BSPJobClient: Current supersteps number: 13 > 11/05/18 16:54:20 INFO bsp.BSPJobClient: Current supersteps number: 14 > 11/05/18 16:54:33 INFO bsp.BSPJobClient: Current supersteps number: 15 > 11/05/18 16:55:18 INFO bsp.BSPJobClient: Current supersteps number: 16 > 11/05/18 16:55:39 INFO bsp.BSPJobClient: Current supersteps number: 17 > 11/05/18 16:55:51 INFO bsp.BSPJobClient: Current supersteps number: 18 > 11/05/18 17:01:33 INFO bsp.BSPJobClient: Current supersteps number: 19 > 11/05/18 17:03:51 INFO bsp.BSPJobClient: Current supersteps number: 20 > 11/05/18 17:04:00 INFO bsp.BSPJobClient: Current supersteps number: 21 > 11/05/18 17:23:09 INFO bsp.BSPJobClient: Current supersteps number: 22 > 11/05/18 17:30:12 INFO bsp.BSPJobClient: Current supersteps number: 23 > 11/05/18 17:30:24 INFO bsp.BSPJobClient: Current supersteps number: 24 > 11/05/18 17:40:01 INFO bsp.BSPJobClient: Current supersteps number: 25 > 11/05/18 17:43:43 INFO bsp.BSPJobClient: Current supersteps number: 26 > 11/05/18 17:43:52 INFO bsp.BSPJobClient: Current supersteps number: 27 > 11/05/18 17:47:25 INFO bsp.BSPJobClient: Current supersteps number: 28 > 11/05/18 17:48:52 INFO bsp.BSPJobClient: Current supersteps number: 29 > 11/05/18 17:49:04 INFO bsp.BSPJobClient: Current supersteps number: 30 > 11/05/18 17:50:28 INFO bsp.BSPJobClient: Current supersteps number: 31 > 11/05/18 17:51:07 INFO bsp.BSPJobClient: Current supersteps number: 32 > 11/05/18 17:51:16 INFO bsp.BSPJobClient: Current supersteps number: 33 > 11/05/18 17:51:52 INFO bsp.BSPJobClient: Current supersteps number: 34 > 11/05/18 17:52:13 INFO bsp.BSPJobClient: Current supersteps number: 35 > 11/05/18 17:52:22 INFO bsp.BSPJobClient: Current supersteps number: 36 > 11/05/18 17:52:40 INFO bsp.BSPJobClient: Current supersteps number: 37 > 11/05/18 17:52:55 INFO bsp.BSPJobClient: Current supersteps number: 38 > 11/05/18 17:53:04 INFO bsp.BSPJobClient: Current supersteps number: 39 > 11/05/18 17:53:16 INFO bsp.BSPJobClient: Current supersteps number: 40 > 11/05/18 17:53:28 INFO bsp.BSPJobClient: Current supersteps number: 41 > 11/05/18 17:53:37 INFO bsp.BSPJobClient: Current supersteps number: 42 > 11/05/18 17:53:49 INFO bsp.BSPJobClient: Current supersteps number: 43 > 11/05/18 17:54:01 INFO bsp.BSPJobClient: Current supersteps number: 44 > 11/05/18 17:54:10 INFO bsp.BSPJobClient: Current supersteps number: 45 > 11/05/18 17:54:22 INFO bsp.BSPJobClient: Current supersteps number: 46 > 11/05/18 17:54:31 INFO bsp.BSPJobClient: Current supersteps number: 47 > 11/05/18 17:54:43 INFO bsp.BSPJobClient: Current supersteps number: 48 > 11/05/18 17:54:52 INFO bsp.BSPJobClient: Current supersteps number: 49 > 11/05/18 17:55:01 INFO bsp.BSPJobClient: Current supersteps number: 50 > 11/05/18 17:55:13 INFO bsp.BSPJobClient: Current supersteps number: 51 > 11/05/18 17:55:25 INFO bsp.BSPJobClient: Current supersteps number: 52 > 11/05/18 17:55:34 INFO bsp.BSPJobClient: Current supersteps number: 53 > 11/05/18 17:55:43 INFO bsp.BSPJobClient: Current supersteps number: 54 > 11/05/18 17:55:55 INFO bsp.BSPJobClient: Current supersteps number: 55 > 11/05/18 17:56:04 INFO bsp.BSPJobClient: Current supersteps number: 56 > 11/05/18 17:56:16 INFO bsp.BSPJobClient: Current supersteps number: 57 > 11/05/18 17:56:25 INFO bsp.BSPJobClient: Current supersteps number: 58 > 11/05/18 17:56:37 INFO bsp.BSPJobClient: Current supersteps number: 59 > 11/05/18 17:56:46 INFO bsp.BSPJobClient: Current supersteps number: 60 > 11/05/18 17:56:58 INFO bsp.BSPJobClient: Current supersteps number: 61 > 11/05/18 17:57:07 INFO bsp.BSPJobClient: Current supersteps number: 62 > 11/05/18 17:57:19 INFO bsp.BSPJobClient: Current supersteps number: 63 > 11/05/18 17:57:28 INFO bsp.BSPJobClient: Current supersteps number: 64 > 11/05/18 17:57:37 INFO bsp.BSPJobClient: Current supersteps number: 65 > 11/05/18 17:57:49 INFO bsp.BSPJobClient: Current supersteps number: 66 > 11/05/18 17:57:58 INFO bsp.BSPJobClient: Current supersteps number: 67 > 11/05/18 17:58:10 INFO bsp.BSPJobClient: Current supersteps number: 68 > 11/05/18 17:58:19 INFO bsp.BSPJobClient: Current supersteps number: 69 > 11/05/18 17:58:25 INFO bsp.BSPJobClient: The total number of supersteps: 69 > Job Finished in 4006.779 seconds > -------------------- RESULTS -------------------- > java.io.IOException: Cannot open filename /user/root/output/_logs > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1497) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1488) > at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:376) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:178) > at > org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1437) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412) > at > org.apache.hama.examples.ShortestPaths.printOutput(ShortestPaths.java:299) > at > org.apache.hama.examples.ShortestPaths.main(ShortestPaths.java:550) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) > at > org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) > at > org.apache.hama.examples.ExampleDriver.main(ExampleDriver.java:34) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hama.util.RunJar.main(RunJar.java:145) > > > On Wed, May 18, 2011 at 5:04 PM, Edward J. Yoon (JIRA) <[email protected]> > wrote: > > > > [ > https://issues.apache.org/jira/browse/HAMA-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035230#comment-13035230] > > > > Edward J. Yoon commented on HAMA-359: > > ------------------------------------- > > > > I tested after changing 'Thread.sleep(100);' to 'Thread.sleep(10000);' at > BSPPeer.sync() method and finally, my job successfully done. > > > > {code} > > > > 2011-05-18 15:15:27,147 INFO org.apache.hadoop.ipc.Server: IPC Server > listener on 40000: starting > > 2011-05-18 15:15:27,149 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 0 on 40000: starting > > 2011-05-18 15:15:27,151 INFO org.apache.hama.bsp.BSPMaster: Starting > RUNNING > > 2011-05-18 15:22:48,062 DEBUG org.apache.hama.bsp.JobInProgress: > numBSPTasks: 16 > > 2011-05-18 15:22:48,065 DEBUG org.apache.hama.bsp.JobInProgress: Job is > initialized. > > 2011-05-18 16:29:27,582 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000005_0' has finished successfully. > > 2011-05-18 16:29:27,583 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000005' has completed. > > 2011-05-18 16:29:27,806 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000004_0' has finished successfully. > > 2011-05-18 16:29:27,806 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000004' has completed. > > 2011-05-18 16:29:28,336 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000015_0' has finished successfully. > > 2011-05-18 16:29:28,336 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000015' has completed. > > 2011-05-18 16:29:28,517 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000000_0' has finished successfully. > > 2011-05-18 16:29:28,517 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000000' has completed. > > 2011-05-18 16:29:28,524 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000013_0' has finished successfully. > > 2011-05-18 16:29:28,524 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000013' has completed. > > 2011-05-18 16:29:28,589 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000012_0' has finished successfully. > > 2011-05-18 16:29:28,589 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000012' has completed. > > 2011-05-18 16:29:28,602 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000001_0' has finished successfully. > > 2011-05-18 16:29:28,602 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000001' has completed. > > 2011-05-18 16:29:28,775 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000014_0' has finished successfully. > > 2011-05-18 16:29:28,775 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000014' has completed. > > 2011-05-18 16:29:28,909 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000010_0' has finished successfully. > > 2011-05-18 16:29:28,909 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000010' has completed. > > 2011-05-18 16:29:28,914 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000007_0' has finished successfully. > > 2011-05-18 16:29:28,914 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000007' has completed. > > 2011-05-18 16:29:28,960 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000006_0' has finished successfully. > > 2011-05-18 16:29:28,960 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000006' has completed. > > 2011-05-18 16:29:29,148 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000011_0' has finished successfully. > > 2011-05-18 16:29:29,148 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000011' has completed. > > 2011-05-18 16:29:29,199 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000008_0' has finished successfully. > > 2011-05-18 16:29:29,199 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000008' has completed. > > 2011-05-18 16:29:29,244 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000009_0' has finished successfully. > > 2011-05-18 16:29:29,244 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000009' has completed. > > 2011-05-18 16:29:29,274 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000002_0' has finished successfully. > > 2011-05-18 16:29:29,274 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000002' has completed. > > 2011-05-18 16:29:29,392 INFO org.apache.hama.bsp.JobInProgress: Taskid > 'attempt_201105181515_0001_000003_0' has finished successfully. > > 2011-05-18 16:29:29,392 INFO org.apache.hama.bsp.TaskInProgress: Task > 'task_201105181515_0001_000003' has completed. > > 2011-05-18 16:29:29,395 DEBUG org.apache.hama.bsp.JobInProgress: Job > successfully done. > > 2011-05-18 16:32:48,365 DEBUG org.apache.hama.bsp.BSPMaster: returns all > jobs: 1 > > {code} > > > >>> Is this still related to the barrier sync? > > > > Yes. the problem related with zk node creation/deletion logic in > enterBarrier() and leaveBarrier() methods. Sometimes they occurs at the same > time. > > > >>> Increasing the timeout won't fix the problem with it. > > > > As i mentioned on chat, JVM garbage collection pause causes zk session > time-out errors. > > > >> Development of Shortest Path Finding Algorithm > >> ---------------------------------------------- > >> > >> Key: HAMA-359 > >> URL: https://issues.apache.org/jira/browse/HAMA-359 > >> Project: Hama > >> Issue Type: New Feature > >> Components: examples > >> Affects Versions: 0.2.0 > >> Reporter: Edward J. Yoon > >> Assignee: Thomas Jungblut > >> Labels: gsoc, gsoc2011, mentor > >> Fix For: 0.3.0 > >> > >> Attachments: HAMA-359-v2.patch, HAMA-359-v3.patch, > HAMA-359-v4.patch, HAMA-359.patch, eddie.patch > >> > >> Original Estimate: 2016h > >> Remaining Estimate: 2016h > >> > >> The goal of this project is development of parallel algorithm for > finding a Shortest Path using Hama BSP. > > > > -- > > This message is automatically generated by JIRA. > > For more information on JIRA, see: > http://www.atlassian.com/software/jira > > > > > > -- > Best Regards, Edward J. Yoon > -- Thomas Jungblut Berlin mobile: 0170-3081070 business: [email protected] private: [email protected]
