----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23340/#review48166 -----------------------------------------------------------
Ship it! Ship It! - Pavan Kumar Athivarapu On July 18, 2014, 9:30 p.m., Sergey Edunov wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23340/ > ----------------------------------------------------------- > > (Updated July 18, 2014, 9:30 p.m.) > > > Review request for giraph. > > > Repository: giraph-git > > > Description > ------- > > Our profiling shows that a lot of apps are neither CPU nor memory or network > bound. Instead they waste a lot of time waiting for lock in MessageStore. > That happens in netty threads. > We should be able to put messages into queue and then process them in other > set of threads. > It has to be configurable because adding another thread level will introduce > additional overhead. > > I introduced new options: > giraph.async.message.store (false by default) that will enable async messaging > giraph.async.message.store.threads (8 by default) that will configure number > of background threads required to process messages. > > > Diffs > ----- > > giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java 036510e > > giraph-core/src/main/java/org/apache/giraph/comm/messages/InMemoryMessageStoreFactory.java > db22503 > > giraph-core/src/main/java/org/apache/giraph/comm/messages/queue/AsyncMessageStoreWrapper.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/comm/messages/queue/PartitionMessage.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/comm/messages/queue/package-info.java > PRE-CREATION > giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java > 3d16e9c > giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java > b2a5c84 > giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java > 0d90a59 > > giraph-core/src/test/java/org/apache/giraph/comm/messages/queue/AsyncMessageStoreWrapperTest.java > PRE-CREATION > giraph-examples/src/test/java/org/apache/giraph/TestCheckpointing.java > 387b937 > > Diff: https://reviews.apache.org/r/23340/diff/ > > > Testing > ------- > > I run pagerank and it gives ~7% improvement over and along with G1 GC it > gives ~15% improvement. And CPU usage is now close to 90% > > > Thanks, > > Sergey Edunov > >
