[
https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903699#comment-15903699
]
Thomas Poepping commented on HIVE-14848:
----------------------------------------
Good point [~spena], doesn't make sense to have this before Hadoop 2.9. Here's
the issue I was facing -- when running against "real S3" using the Tez
execution engine, the credentials passed via hive-site.xml are not passed to
individual Tez workers, which fails the tests. There are ways around this, and
myself and my team have successfully worked around it.
Honestly, if nobody else has any issue, I'm happy to wait on this for Hadoop
2.9.0 as well. I'll try to track on my end so we don't drop this issue by
accident.
> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
> ------------------------------------------------------------------------
>
> Key: HIVE-14848
> URL: https://issues.apache.org/jira/browse/HIVE-14848
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 2.2.0
> Reporter: Sergio Peña
> Assignee: Sergio Peña
> Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to
> read
> data from S3 cannot use them because S3 values are stripped from the Job
> configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
> driverContext) {
> ...
> conf.stripHiddenConfigurations(job);
> this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR
> {{mapreduce.job.redacted-properties}} that can be used to hide this list on
> the MR side (such as history server UI) to allow MR run the job without
> issues.
> UPDATE:
> Change the call to stripHiddenConfigurations() in
> ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken
> for running hive-blobstore suite against Tez
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
