zhangbutao commented on PR #5257: URL: https://github.com/apache/hive/pull/5257#issuecomment-2167670614
> @zhangbutao, 1 question here, when I try to insert the data using tez, the query fails with NoAuthWithAWSException on staging dir and when I debugged it, I found that the conf is not passed to tez (on yarn) container because of > > https://github.com/apache/hive/blob/33cadc5b498b57f779cddc9bf4e3f8aef2d9f6dc/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1478 > > > As fs.s3a.secret.key and fs.s3a.access.key exist in default hidden list. > I have tried it on hive3 as I don't have a setup of hive4 with S3. **_I understand that because of patch differences, issue might not occur with hive4 but in hive4 also, this DAGUtils is removing those s3 credentials from jobconf (tested on local setup with hdfs based table)_**. > If you have any insights, can you help me understand if I am missing anything? The same happens in select query as well. In hive3 I have tested by removing the fs.s3a* configs from hidden list, then the insert/select query is passing. > > Also, do we want to add metaconf:fs.s3a.* configs to hidden list? As in the above attached screenshot, I am able to see the credential value. @Aggarwal-Raghav I think this is a good question. See previous ticket https://issues.apache.org/jira/browse/HIVE-25929. At present, if we want to propagate the s3 related properttity to tez, we need to remove the fs.s3a* configs from hidden list, but this maybe insecurely. I just did a magic way which only set a specific hidden property to allow the fs.s3a* configs to be propagated to tez. Like this: ``` <property> <name>hive.conf.hidden.list</name> <value>javax.jdo.option.ConnectionPassword</value> </property> ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
