-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4702/
-----------------------------------------------------------
Review request for hcatalog.
Summary
-------
Cause:
FosterStorageHandler sorts the partition keys and creates the output
directories in that order instead of the actual order. Setting file permissions
in FileOutputCommitterContainer was failing as the path constructed was in the
order partition keys were defined and that path did not exist.
for(FieldSchema partKey : table.getPartitionKeys()){
partPath = constructPartialPartPath(partPath,
partKey.getName().toLowerCase(), partKVs);
fs.setPermission(partPath, perms);
}
Because of the exception, the partitions were never registered with hcat
server. When running through PIG this exception was gobbled because of PIG-2568
(PigOutputCommitter hide exception in commitJob) and so the map job was
successful. The only stack trace that appears in the job is HCAT-236 which is
kind of misleading as it is just a logged message.
So basically if the partition keys were not defined in the sorted order, it
would fail.
Fix:
Changed FosterStorageHandler to construct the output path same way as
FileOutputCommitterContainer.
This addresses bug HCATALOG-367.
https://issues.apache.org/jira/browse/HCATALOG-367
Diffs
-----
http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FosterStorageHandler.java
1311907
Diff: https://reviews.apache.org/r/4702/diff
Testing
-------
Unit tests pass. Did an integration test issuing pig commands. e2e tests not
run.
Thanks,
Rohini
