----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14068/ -----------------------------------------------------------
Review request for giraph. Repository: giraph-git Description ------- This patch provides a way to use Rexster as output format endpoint. By means of ZooKeeper coordination, the workers will first write all the vertices and then all the edges. Such an order is required by the bluprint API. Limitations: - The current approach does not yet work with neo4j. In fact, neo4j does not properly handle saving vertices not in order of ID. I am looking into this currently. - This approach does not scale. In fact, the batch-kibble provided by Rexster and used by the output format loads the whole graph in memory before committing the data collected into the underlining database. This will be solved with a Giraph provided kibble to stream the data from the output format. Diffs ----- giraph-rexster/src/main/java/org/apache/giraph/rexster/conf/GiraphRexsterConstants.java c4b4655 giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterEdgeInputFormat.java 4dee078 giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterEdgeOutputFormat.java PRE-CREATION giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterVertexInputFormat.java da45b5b giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterVertexOutputFormat.java PRE-CREATION giraph-rexster/src/main/java/org/apache/giraph/rexster/io/formats/RexsterLongDoubleFloatEdgeOutputFormat.java PRE-CREATION giraph-rexster/src/main/java/org/apache/giraph/rexster/io/formats/RexsterLongDoubleFloatVertexOutputFormat.java PRE-CREATION giraph-rexster/src/main/java/org/apache/giraph/rexster/utils/RexsterUtils.java e669ca9 giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestAbstractRexsterInputFormat.java 09b27d3 giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestRexsterLongDoubleFloatInputFormat.java PRE-CREATION giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestRexsterLongDoubleFloatOutputFormat.java PRE-CREATION giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestRexsterLongDoubleFloatVertexInputFormat.java a68db67 Diff: https://reviews.apache.org/r/14068/diff/ Testing ------- tested on a small cluster using tinkergraph as database exported with Rexster. Thanks, Armando
