----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23340/ -----------------------------------------------------------
(Updated July 17, 2014, 8:21 p.m.) Review request for giraph. Changes ------- Addressing CR issues Repository: giraph-git Description ------- Our profiling shows that a lot of apps are neither CPU nor memory or network bound. Instead they waste a lot of time waiting for lock in MessageStore. That happens in netty threads. We should be able to put messages into queue and then process them in other set of threads. It has to be configurable because adding another thread level will introduce additional overhead. I introduced new options: giraph.async.message.store (false by default) that will enable async messaging giraph.async.message.store.threads (8 by default) that will configure number of background threads required to process messages. Diffs (updated) ----- giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java b3f8733 giraph-core/src/main/java/org/apache/giraph/comm/messages/InMemoryMessageStoreFactory.java f691d3e giraph-core/src/main/java/org/apache/giraph/comm/messages/queue/AsyncMessageStoreWrapper.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/comm/messages/queue/PartitionMessage.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/comm/messages/queue/package-info.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 7d7ceb2 giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java e13eedd giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java dbe6a45 giraph-core/src/test/java/org/apache/giraph/comm/messages/queue/AsyncMessageStoreWrapperTest.java PRE-CREATION Diff: https://reviews.apache.org/r/23340/diff/ Testing ------- I run pagerank and it gives ~7% improvement over and along with G1 GC it gives ~15% improvement. And CPU usage is now close to 90% Thanks, Sergey Edunov
