[
https://issues.apache.org/jira/browse/HCATALOG-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449780#comment-13449780
]
Travis Crawford commented on HCATALOG-451:
------------------------------------------
[~toffer] I totally agree the Pig version issue is something we have to fix. We
have some internal patches that can't go into trunk until that happens (such as
maps with schemas, boolean support, ...) and are eager to help if possible.
That said, I'm concerned about code rot because the Pig issue is not a quick
fix, and future changes could lead to having to redo this patch which is never
fun. I agree with [~rohini] that we can file a new bug to fix the known issue
with this patch, and set it as blocked by the Pig 0.10.0 upgrade.
Thoughts? We definitely need to address the antlr version mismatch that's
preventing us from upgrading Pig, but I'm not sure it should hold up this
patch, provided we file a new issue about the known issue.
> Partitions are created even when Jobs are aborted
> -------------------------------------------------
>
> Key: HCATALOG-451
> URL: https://issues.apache.org/jira/browse/HCATALOG-451
> Project: HCatalog
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.4, 0.5
> Environment: Hadoop 1.0.2, non-dynamic partitions.
> Reporter: Mithun Radhakrishnan
> Assignee: Vandana Ayyalasomayajula
> Fix For: 0.4.1
>
> Attachments: HCAT-451-trunk.02.patch, HCATALOG-451.0.patch,
> HCATALOG-451-branch-0.4.02.patch, HCATALOG-451-branch-0.4.03.patch,
> HCATALOG-451-branch-0.4.patch
>
>
> If an MR job using HCatOutputFormat fails, and
> FileOutputCommitterContainer::abortJob() is called, one would expect that
> partitions aren't created/registered with HCatalog.
> When using dynamic-partitions, one sees that this behaves correctly. But when
> static-partitions are used, partitions are created regardless of whether the
> Job succeeded or failed.
> (This manifested as a failure when the job is repeated. The retry-job fails
> to launch since the partitions already exist from the last failed run.)
> This is a result of bad code in FileOutputCommitter::cleanupJob(), which
> seems to do an unconditional partition-add. This can be fixed by adding a
> check for the output directory before adding partitions (in the
> !dynamicParititoning case), since the directory is removed in abortJob().
> We'll have a patch for this shortly. As an aside, we ought to move the
> partition-creation into commitJob(), where it logically belongs. cleanupJob()
> is deprecated and common to both success and failure code paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
