-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4702/
-----------------------------------------------------------
(Updated 2012-04-12 00:50:11.777910)
Review request for hcatalog.
Changes
-------
Updated patch with modification to testcases in TestHCatPartitioned.java to
test this scenario.
Verified that the tests fail with "org.apache.hcatalog.common.HCatException :
2006 : Error adding partition to metastore. Cause : java.io.IOException: Failed
to set permissions of path:
/user/hive/warehouse/testhcatpartitionedtable/part1=p1value1 to 0755" with the
earlier FosterStorageHandler that sorts partition columns.
Summary
-------
Cause:
FosterStorageHandler sorts the partition keys and creates the output
directories in that order instead of the actual order. Setting file permissions
in FileOutputCommitterContainer was failing as the path constructed was in the
order partition keys were defined and that path did not exist.
for(FieldSchema partKey : table.getPartitionKeys()){
partPath = constructPartialPartPath(partPath,
partKey.getName().toLowerCase(), partKVs);
fs.setPermission(partPath, perms);
}
Because of the exception, the partitions were never registered with hcat
server. When running through PIG this exception was gobbled because of PIG-2568
(PigOutputCommitter hide exception in commitJob) and so the map job was
successful. The only stack trace that appears in the job is HCAT-236 which is
kind of misleading as it is just a logged message.
So basically if the partition keys were not defined in the sorted order, it
would fail.
Fix:
Changed FosterStorageHandler to construct the output path same way as
FileOutputCommitterContainer.
This addresses bug HCATALOG-367.
https://issues.apache.org/jira/browse/HCATALOG-367
Diffs (updated)
-----
http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FosterStorageHandler.java
1325041
http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/HCatMapReduceTest.java
1325041
http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestHCatPartitioned.java
1325041
Diff: https://reviews.apache.org/r/4702/diff
Testing
-------
Unit tests pass. Did an integration test issuing pig commands. e2e tests not
run.
Thanks,
Rohini
