[
https://issues.apache.org/jira/browse/HCATALOG-367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252149#comment-13252149
]
[email protected] commented on HCATALOG-367:
--------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4702/#review6866
-----------------------------------------------------------
Ship it!
Looks good. Unit tests and e2e tests (except hbase, need to fix my env) passes.
- Francis
On 2012-04-12 00:50:11, Rohini Palaniswamy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4702/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-12 00:50:11)
bq.
bq.
bq. Review request for hcatalog.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Cause:
bq. FosterStorageHandler sorts the partition keys and creates the output
directories in that order instead of the actual order. Setting file permissions
in FileOutputCommitterContainer was failing as the path constructed was in the
order partition keys were defined and that path did not exist.
bq. for(FieldSchema partKey : table.getPartitionKeys()){
bq. partPath = constructPartialPartPath(partPath,
partKey.getName().toLowerCase(), partKVs);
bq. fs.setPermission(partPath, perms);
bq.
bq. }
bq.
bq. Because of the exception, the partitions were never registered with hcat
server. When running through PIG this exception was gobbled because of PIG-2568
(PigOutputCommitter hide exception in commitJob) and so the map job was
successful. The only stack trace that appears in the job is HCAT-236 which is
kind of misleading as it is just a logged message.
bq.
bq. So basically if the partition keys were not defined in the sorted order,
it would fail.
bq.
bq. Fix:
bq. Changed FosterStorageHandler to construct the output path same way as
FileOutputCommitterContainer.
bq.
bq.
bq. This addresses bug HCATALOG-367.
bq. https://issues.apache.org/jira/browse/HCATALOG-367
bq.
bq.
bq. Diffs
bq. -----
bq.
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FosterStorageHandler.java
1325041
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/HCatMapReduceTest.java
1325041
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestHCatPartitioned.java
1325041
bq.
bq. Diff: https://reviews.apache.org/r/4702/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Unit tests pass. Did an integration test issuing pig commands. e2e tests
not run.
bq.
bq.
bq. Thanks,
bq.
bq. Rohini
bq.
bq.
> Pig writes data but partition information isn't getting updated
> ---------------------------------------------------------------
>
> Key: HCATALOG-367
> URL: https://issues.apache.org/jira/browse/HCATALOG-367
> Project: HCatalog
> Issue Type: Bug
> Components: pig
> Affects Versions: 0.4
> Reporter: David Capwell
> Assignee: Rohini Palaniswamy
> Attachments: HCATALOG-367-1.patch, HCATALOG-367.patch
>
>
> I see that when ever pig runs into HCATALOG-236 that the data on HDFS is
> written but the hive partitions are not getting updated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
