-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12377/
-----------------------------------------------------------
(Updated July 10, 2013, 6:01 p.m.)
Review request for giraph.
Changes
-------
Nitay's comments
Bugs: GIRAPH-713
https://issues.apache.org/jira/browse/GIRAPH-713
Repository: giraph-git
Description
-------
In some cases, network is much slower than all the computation stuff we do, and
we could benefit from compressing requests.
I am using LZ4 compression. Tried out two versions of Snappy and some java
ones, and LZ4 had the least CPU overhead, and about the same speed as Snappy.
It's also easy to plug in your own compressor. In cases when we are not bounded
by network, CPU overhead of using compression is about 10% (testing with
PageRankBenchmark) and time stays about the same. Depending on how slow your
network is and how good your data compresses this can lead to big time savings.
Also added compression to metrics, from one of the pagerank iterations:
https://gist.github.com/majakabiljo/5962269
Diffs (updated)
-----
giraph-core/pom.xml cab0157
giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/LZ4RequestCodec.java
PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/NoOpRequestCodec.java
PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestCodec.java
PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestDecoder.java
6eb6549
giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java
83b408e
giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 6655834
giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c4cc96f
giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
ed63192
giraph-core/src/main/java/org/apache/giraph/metrics/GiraphMetrics.java
7d980ea
giraph-core/src/main/java/org/apache/giraph/metrics/MetricNames.java cc237ac
giraph-core/src/main/java/org/apache/giraph/utils/Sizes.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayInputStream.java
20ed92b
giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java
9ff1242
giraph-core/src/test/java/org/apache/giraph/comm/LZ4RequestCodecTest.java
PRE-CREATION
pom.xml 72fe6f2
Diff: https://reviews.apache.org/r/12377/diff/
Testing
-------
mvn clean verify
PageRank on cluster
Added test for compressor
Verified that NoOpCompressor has the same performance as before
Thanks,
Maja Kabiljo
