----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23340/ -----------------------------------------------------------
(Updated July 18, 2014, 9:30 p.m.) Review request for giraph. Changes ------- Merged from trunk, verified that checkpointing still works and performance gains a still there. Repository: giraph-git Description ------- Our profiling shows that a lot of apps are neither CPU nor memory or network bound. Instead they waste a lot of time waiting for lock in MessageStore. That happens in netty threads. We should be able to put messages into queue and then process them in other set of threads. It has to be configurable because adding another thread level will introduce additional overhead. I introduced new options: giraph.async.message.store (false by default) that will enable async messaging giraph.async.message.store.threads (8 by default) that will configure number of background threads required to process messages. Diffs (updated) ----- giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java 036510e giraph-core/src/main/java/org/apache/giraph/comm/messages/InMemoryMessageStoreFactory.java db22503 giraph-core/src/main/java/org/apache/giraph/comm/messages/queue/AsyncMessageStoreWrapper.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/comm/messages/queue/PartitionMessage.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/comm/messages/queue/package-info.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 3d16e9c giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java b2a5c84 giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 0d90a59 giraph-core/src/test/java/org/apache/giraph/comm/messages/queue/AsyncMessageStoreWrapperTest.java PRE-CREATION giraph-examples/src/test/java/org/apache/giraph/TestCheckpointing.java 387b937 Diff: https://reviews.apache.org/r/23340/diff/ Testing ------- I run pagerank and it gives ~7% improvement over and along with G1 GC it gives ~15% improvement. And CPU usage is now close to 90% Thanks, Sergey Edunov
