-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9138/
-----------------------------------------------------------
Review request for giraph.
Description
-------
Adds status updates to saving of the vertices and improve the overall logic of
when to print status of loading/storing the graph (every 250k vertices or 15
secs). This will help us to see which are the slow workers when saving output.
This updates the Hadoop status messages as well as prints to the task log. I
also made this consistent for the loading.
Task log messages look like the following:
INFO 2013-01-29 12:51:46,044 [main]
org.apache.giraph.worker.BspServiceWorker - saveVertices: Saved 98751 out of
1000000 vertices, on partition 2 out of 24
INFO 2013-01-29 12:52:25,539 [main]
org.apache.giraph.worker.BspServiceWorker - saveVertices: Saved 348752 out of
1000000 vertices, on partition 8 out of 24
INFO 2013-01-29 12:53:28,062 [main]
org.apache.giraph.worker.BspServiceWorker - saveVertices: Saved 598753 out of
1000000 vertices, on partition 14 out of 24
I added an option for dumping output to PageRankBenchmark to test this as well.
This addresses bug GIRAPH-492.
https://issues.apache.org/jira/browse/GIRAPH-492
Diffs
-----
giraph-core/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java
3ef471a711183dd147990c5f6bb07485a58f5a71
giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedService.java
83fba57af3f11f6c412ec59e2d121e57ea280d98
giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java
399dc72896673fbf75d2b7b933c4ea02a08f25ea
giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java
294c2c71017b5273240c9dce1f3de2be65aed289
giraph-core/src/main/java/org/apache/giraph/graph/FinishedSuperstepStats.java
d888d1038026c83b5f82956b6872eb64a44dd700
giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java
401e07bb346e8ac43600992718f1f231db85aa7c
giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java
7ad290244946b89c6a563a86536d8e898e1c0aec
giraph-core/src/main/java/org/apache/giraph/utils/LoggerUtils.java
81dfd1d8d9b27442543db1259235fb1825f72f7e
giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java
d5ad62b39517f06884ef1c003c5e91348cbe2459
giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java
7522027b7306aff17668e17f4895a39e23e8a590
Diff: https://reviews.apache.org/r/9138/diff/
Testing
-------
Passsed unittests and tested on a real cluster with PageRankBenchmark.
Thanks,
Avery Ching
