[
https://issues.apache.org/jira/browse/HAMA-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edward J. Yoon updated HAMA-642:
--------------------------------
Fix Version/s: (was: 0.6.0)
Job never finishes.
I doubt there's a JDBM's bug (concurrent access or structural bug might be a
part of the problem) or bug in this patch.
{code}
[oracle@bda01 hama-0.6.0-MEM]$ bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar
sssp 3 /user/oracle/edward/rangraph/part-r-00000 testout
12/09/23 19:20:12 INFO bsp.FileInputFormat: Total input paths to process : 1
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node:
/switch1/192.168.10.12:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node:
/switch1/192.168.10.14:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node:
/switch1/192.168.10.7:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node:
/switch1/192.168.10.8:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node:
/switch1/192.168.10.3:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node:
/switch1/192.168.10.2:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node:
/switch1/192.168.10.17:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node:
/switch1/192.168.10.18:50010
12/09/23 19:20:12 INFO net.NetworkTopology: Adding a new node:
/switch1/192.168.10.5:50010
12/09/23 19:20:12 INFO bsp.FileInputFormat: Total # of splits: 85
12/09/23 19:20:12 INFO bsp.BSPJobClient: Running job: job_201209231919_0001
12/09/23 19:20:15 INFO bsp.BSPJobClient: Current supersteps number: 0
12/09/23 19:20:18 INFO bsp.BSPJobClient: Current supersteps number: 2
12/09/23 19:20:27 INFO bsp.BSPJobClient: Current supersteps number: 3
12/09/23 19:20:36 INFO bsp.BSPJobClient: Current supersteps number: 8
12/09/23 19:20:39 INFO bsp.BSPJobClient: Current supersteps number: 10
12/09/23 19:20:42 INFO bsp.BSPJobClient: Current supersteps number: 12
12/09/23 19:20:48 INFO bsp.BSPJobClient: Current supersteps number: 13
12/09/23 19:21:00 INFO bsp.BSPJobClient: Current supersteps number: 14
12/09/23 19:21:06 INFO bsp.BSPJobClient: Current supersteps number: 15
12/09/23 19:21:15 INFO bsp.BSPJobClient: Current supersteps number: 16
12/09/23 19:21:21 INFO bsp.BSPJobClient: Current supersteps number: 17
12/09/23 19:21:30 INFO bsp.BSPJobClient: Current supersteps number: 18
12/09/23 19:21:36 INFO bsp.BSPJobClient: Current supersteps number: 19
12/09/23 19:21:42 INFO bsp.BSPJobClient: Current supersteps number: 20
12/09/23 19:21:48 INFO bsp.BSPJobClient: Current supersteps number: 21
12/09/23 19:21:54 INFO bsp.BSPJobClient: Current supersteps number: 22
12/09/23 19:21:57 INFO bsp.BSPJobClient: Current supersteps number: 23
12/09/23 19:22:03 INFO bsp.BSPJobClient: Current supersteps number: 24
12/09/23 19:22:06 INFO bsp.BSPJobClient: Current supersteps number: 26
12/09/23 19:22:09 INFO bsp.BSPJobClient: Current supersteps number: 28
12/09/23 19:22:12 INFO bsp.BSPJobClient: Current supersteps number: 31
12/09/23 19:22:15 INFO bsp.BSPJobClient: Current supersteps number: 33
12/09/23 19:22:18 INFO bsp.BSPJobClient: Current supersteps number: 37
12/09/23 19:22:21 INFO bsp.BSPJobClient: Current supersteps number: 40
12/09/23 19:22:24 INFO bsp.BSPJobClient: Current supersteps number: 48
12/09/23 19:22:27 INFO bsp.BSPJobClient: Current supersteps number: 49
12/09/23 19:22:27 INFO bsp.BSPJobClient: The total number of supersteps: 49
12/09/23 19:22:27 INFO bsp.BSPJobClient: Counters: 8
12/09/23 19:22:27 INFO bsp.BSPJobClient:
org.apache.hama.bsp.JobInProgress$JobCounter
12/09/23 19:22:27 INFO bsp.BSPJobClient: LAUNCHED_TASKS=85
12/09/23 19:22:27 INFO bsp.BSPJobClient:
org.apache.hama.bsp.BSPPeerImpl$PeerCounter
12/09/23 19:22:27 INFO bsp.BSPJobClient: SUPERSTEPS=49
12/09/23 19:22:27 INFO bsp.BSPJobClient: SUPERSTEP_SUM=4165
12/09/23 19:22:27 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=1692428
12/09/23 19:22:27 INFO bsp.BSPJobClient: IO_BYTES_READ=1167771205
12/09/23 19:22:27 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=1157989750
12/09/23 19:22:27 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=10000000
12/09/23 19:22:27 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=578994875
Job Finished in 135.463 seconds
----
[oracle@bda01 hama-0.6.0-JDBM]$ bin/hama jar
../hama-0.6.0-MEM/hama-examples-0.6.0-SNAPSHOT.jar sssp 3
/user/oracle/edward/rangraph/part-r-00000 testout
....
12/09/23 20:43:48 INFO bsp.BSPJobClient: Current supersteps number: 485
12/09/23 20:44:00 INFO bsp.BSPJobClient: Current supersteps number: 486
12/09/23 20:44:09 INFO bsp.BSPJobClient: Current supersteps number: 487
12/09/23 20:44:18 INFO bsp.BSPJobClient: Current supersteps number: 488
12/09/23 20:44:30 INFO bsp.BSPJobClient: Current supersteps number: 489
12/09/23 20:44:39 INFO bsp.BSPJobClient: Current supersteps number: 490
12/09/23 20:44:48 INFO bsp.BSPJobClient: Current supersteps number: 491
12/09/23 20:45:00 INFO bsp.BSPJobClient: Current supersteps number: 492
12/09/23 20:45:09 INFO bsp.BSPJobClient: Current supersteps number: 493
12/09/23 20:45:18 INFO bsp.BSPJobClient: Current supersteps number: 494
12/09/23 20:45:30 INFO bsp.BSPJobClient: Current supersteps number: 495
12/09/23 20:45:39 INFO bsp.BSPJobClient: Current supersteps number: 496
12/09/23 20:45:48 INFO bsp.BSPJobClient: Current supersteps number: 497
12/09/23 20:46:00 INFO bsp.BSPJobClient: Current supersteps number: 498
12/09/23 20:46:09 INFO bsp.BSPJobClient: Current supersteps number: 499
12/09/23 20:46:18 INFO bsp.BSPJobClient: Current supersteps number: 500
12/09/23 20:46:30 INFO bsp.BSPJobClient: Current supersteps number: 501
^C[oracle@bda01 hama-0.6.0-JDBM]$
[oracle@bda01 hama-0.6.0-JDBM]$ bin/hama job -list all
1 jobs submitted
States are:
Running : 1 Succeded : 2 Failed : 3 Prep : 4
JobId State StartTime UserName
job_201209231923_0001 1 1348453426073 oracle
[oracle@bda01 hama-0.6.0-JDBM]$ bin/hama job -kill job_201209231923_0001
Killed job job_201209231923_0001
{code}
> Make GraphRunner disk based
> ---------------------------
>
> Key: HAMA-642
> URL: https://issues.apache.org/jira/browse/HAMA-642
> Project: Hama
> Issue Type: Improvement
> Components: graph
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Attachments: HAMA-642_unix_1.patch, HAMA-642_unix_2.patch,
> HAMA-scale_1.patch, HAMA-scale_2.patch, HAMA-scale_3.patch, HAMA-scale_4.patch
>
>
> To improve scalability we can improve the graph runner to be disk based.
> Which basically means:
> - We have just a single Vertex instance that get's refilled.
> - We directly write vertices to disk after partitioning
> - In every superstep we iterate over the vertices on disk, fill the vertex
> instance and call the users compute functions
> Problems:
> - State other than vertex value can't be stored easy
> - How do we deal with random access after messages have arrived?
> So I think we should make the graph runner more hybrid, like using the queues
> we have implemented in the messaging. So the graphrunner can be configured to
> run completely on disk, in cached mode or in in-memory mode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
