[
https://issues.apache.org/jira/browse/HADOOP-19091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876989#comment-17876989
]
Steve Loughran commented on HADOOP-19091:
-----------------------------------------
don't think anyone has ever tested that. it's not something you can use in
production unless you have the profiles everywhere *or* you set up delegation
tokens to collect full/session credentials.
I'd recommend you group the cloudstore jar and use its storediag command to see
what it thinks: https://github.com/steveloughran/cloudstore
email me direct with the results at stevel@ apache unless you can identify any
obvious problem. it may be that the provider doesn't work or there's some
integration issue.
if you are being paranoid about keeping secrets off the source tree -I
appeciate the caution. i keep all my settings in a different directory managed
as a local git repo (no remote backup anywhere) and use XInclude to pull it in.
lets me keep things in SCM, share the same config across multiple copies of the
source tree, etc, etc.
just add an xinclude reference into the src/test/resources/auth-keys.xml
{code}
<include xmlns="http://www.w3.org/2001/XInclude";
href="///home/stevel/..../secrets/auth-keys.xml" >
</include>
{code}
> Add support for Tez to MagicS3GuardCommitter
> --------------------------------------------
>
> Key: HADOOP-19091
> URL: https://issues.apache.org/jira/browse/HADOOP-19091
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 3.3.6
> Environment: Pig 17/Hive 3.1.3 with Hadoop 3.3.3 on AWS EMR 6-12.0
> Reporter: Venkatasubrahmanian Narayanan
> Assignee: Venkatasubrahmanian Narayanan
> Priority: Major
> Attachments: 0001-AWS-Hive-Changes.patch,
> 0002-HIVE-27698-Backport-of-HIVE-22398-Remove-legacy-code.patch,
> HADOOP-19091-HIVE-WIP.patch
>
>
> The MagicS3GuardCommitter assumes that the JobID of the task is the same as
> that of the job's application master when writing/reading the .pendingset
> file. This assumption is not valid when running with Tez, which creates
> slightly different JobIDs for tasks and the application master.
>
> While the MagicS3GuardCommitter is intended only for MRv2, it mostly works
> fine with an MRv1 wrapper with Hive/Pig (with some minor changes to Hive) run
> in MR mode. This issue only crops up when running queries with the Tez
> execution engine. I can upload a patch to Hive 3.1 to reproduce this error on
> EMR if needed.
>
> Fixing this will probably require work from both Tez and Hadoop, wanted to
> start a discussion here so we can figure out how exactly we go about this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
