[
https://issues.apache.org/jira/browse/HADOOP-19091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821372#comment-17821372
]
Venkatasubrahmanian Narayanan edited comment on HADOOP-19091 at 2/27/24 7:34 PM:
---------------------------------------------------------------------------------
[~srahman] I've uploaded my WIP Hive patch (there are a couple of other open
sourced patches which need to be backported to Hive 3.1 that I've uploaded as
well). I still need to clean up a couple of things (hence why the patch
hardcodes an expectation that tables are on S3), but the basic idea is to add
an MRv1 wrapper of the MagicS3GuardCommitter similar to how the
FileOutputCommitter for MRv1 is implemented, and since Hive uses MRv1 it only
requires incidental changes to treat paths the way the magic committer expects.
was (Author: vnarayanan7):
[~srahman] I've uploaded my WIP Hive patch (there are a couple of other open
sourced patches which need to be backported to Hive 3.1 that I've uploaded as
well). I still need to clean up a couple of things (hence why the patch
hardcodes an expectation that tables are on. S3), but the basic idea is to add
an MRv1 wrapper of the MagicS3GuardCommitter similar to how the
FileOutputCommitter for MRv1 is implemented, and since Hive uses MRv1 it only
requires incidental changes to treat paths the way the magic committer expects.
> Add support for Tez to MagicS3GuardCommitter
> --------------------------------------------
>
> Key: HADOOP-19091
> URL: https://issues.apache.org/jira/browse/HADOOP-19091
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 3.4.0, 3.3.6
> Environment: Pig 17/Hive 3.1.3 with Hadoop 3.3.3 on AWS EMR 6-12.0
> Reporter: Venkatasubrahmanian Narayanan
> Priority: Major
> Attachments: 0001-AWS-Hive-Changes.patch,
> 0002-HIVE-27698-Backport-of-HIVE-22398-Remove-legacy-code.patch,
> HADOOP-19091-HIVE-WIP.patch
>
>
> The MagicS3GuardCommitter assumes that the JobID of the task is the same as
> that of the job's application master when writing/reading the .pendingset
> file. This assumption is not valid when running with Tez, which creates
> slightly different JobIDs for tasks and the application master.
>
> While the MagicS3GuardCommitter is intended only for MRv2, it mostly works
> fine with an MRv1 wrapper with Hive/Pig (with some minor changes to Hive) run
> in MR mode. This issue only crops up when running queries with the Tez
> execution engine. I can upload a patch to Hive 3.1 to reproduce this error on
> EMR if needed.
>
> Fixing this will probably require work from both Tez and Hadoop, wanted to
> start a discussion here so we can figure out how exactly we go about this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
