-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8611/
-----------------------------------------------------------
(Updated Dec. 15, 2012, 1:23 a.m.)
Review request for giraph.
Description (updated)
-------
GIRAPH-453: Pure Hive I/O (nitay)
For now this is only the Input side of things. One particular thing I added was
the concept of "profiles", allowing for easily reading from multiple tables.
This should remove a lot of the cruft around the GiraphHCat* classes.
Note in the diff I separated the code so that there would be a Giraph-unrelated
Hive-only portion (under package org.apache.hadoop.hive). Things under this
package (and its children) do not touch any Giraph code, and so can be
contributed as an IOFormat back to Hive itself.
In terms of performance this is on par with our current HCatalog stuff. I ran a
few jobs and noticed at most a few seconds of difference between the input
supersteps.
This addresses bug GIRAPH-453.
https://issues.apache.org/jira/browse/GIRAPH-453
Diffs
-----
giraph-formats-contrib/pom.xml 9326c28420be2749fa724ebd86d9555ae209a2ee
giraph-formats-contrib/src/main/java/org/apache/giraph/io/hcatalog/HCatGiraphRunner.java
PRE-CREATION
giraph-formats-contrib/src/main/java/org/apache/giraph/io/hcatalog/HiveGiraphRunner.java
7a7c2f87bc50eaebe879e2eff2da661b504096b0
giraph-hive/pom.xml PRE-CREATION
giraph-hive/src/main/assembly/compile.xml PRE-CREATION
giraph-hive/src/main/java/org/apache/giraph/hive/HiveConstants.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/giraph/hive/HiveEdgeInputFormat.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/giraph/hive/HiveEdgeReader.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/giraph/hive/HiveUtils.java PRE-CREATION
giraph-hive/src/main/java/org/apache/giraph/hive/HiveVertexCreator.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/giraph/hive/HiveVertexInputFormat.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/giraph/hive/HiveVertexReader.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/giraph/hive/VertexCreator.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/HiveApiInputFormat.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/HiveApiOutputFormat.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/HiveInputDescription.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/HiveOutputDescription.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/HiveRecord.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/common/Classes.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/common/HadoopUtils.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/common/ProgressReporter.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/common/SerDes.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/common/Writables.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/ApiInputSplit.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/ApiRecord.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/ApiRecordReader.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/InputConf.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/InputPartition.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/InputPartitions.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/InputSplitData.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/ListParser.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/MapParser.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/ObjectParser.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/Parsers.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/PrimitiveParser.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/StructParser.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/UnionParser.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/output/OutputConf.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedField.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedList.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedMap.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedPrimitive.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedSchema.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedStruct.java
PRE-CREATION
giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedUnion.java
PRE-CREATION
pom.xml 47736351c45f4cdb55df762506069eaed7126a7e
Diff: https://reviews.apache.org/r/8611/diff/
Testing
-------
Thanks,
Nitay Joffe
