----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4702/#review6866 -----------------------------------------------------------
Ship it! Looks good. Unit tests and e2e tests (except hbase, need to fix my env) passes. - Francis On 2012-04-12 00:50:11, Rohini Palaniswamy wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/4702/ > ----------------------------------------------------------- > > (Updated 2012-04-12 00:50:11) > > > Review request for hcatalog. > > > Summary > ------- > > Cause: > FosterStorageHandler sorts the partition keys and creates the output > directories in that order instead of the actual order. Setting file > permissions in FileOutputCommitterContainer was failing as the path > constructed was in the order partition keys were defined and that path did > not exist. > for(FieldSchema partKey : table.getPartitionKeys()){ > partPath = constructPartialPartPath(partPath, > partKey.getName().toLowerCase(), partKVs); > fs.setPermission(partPath, perms); > > } > > Because of the exception, the partitions were never registered with hcat > server. When running through PIG this exception was gobbled because of > PIG-2568 (PigOutputCommitter hide exception in commitJob) and so the map job > was successful. The only stack trace that appears in the job is HCAT-236 > which is kind of misleading as it is just a logged message. > > So basically if the partition keys were not defined in the sorted order, it > would fail. > > Fix: > Changed FosterStorageHandler to construct the output path same way as > FileOutputCommitterContainer. > > > This addresses bug HCATALOG-367. > https://issues.apache.org/jira/browse/HCATALOG-367 > > > Diffs > ----- > > > http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FosterStorageHandler.java > 1325041 > > http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/HCatMapReduceTest.java > 1325041 > > http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestHCatPartitioned.java > 1325041 > > Diff: https://reviews.apache.org/r/4702/diff > > > Testing > ------- > > Unit tests pass. Did an integration test issuing pig commands. e2e tests not > run. > > > Thanks, > > Rohini > >
