Steve Loughran created HADOOP-16567:
---------------------------------------
Summary: S3A Secret access to fall back to XML if credential
provider raises IOE.
Key: HADOOP-16567
URL: https://issues.apache.org/jira/browse/HADOOP-16567
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.1.2
Reporter: Steve Loughran
This is hive related. Hive can put secrets into a JCEKS file which only hive
may read.
S3A file systems created on behalf of a user do not have access to this file.
Yet it is listed as the credential provider in the hadoop.credential.providers
option in core-site -and that is marked as final. When the S3 a initializre()
method looks up passwords and encryption keys, the failure to open the file
raises an IOE -and the FS cannot be instantiated.
Proposed: {{S3AUtils.lookupPassword()}} to catch such exceptions, and fall back
to using {{Configuration.get()}} and so retrieve any property in the XML. If
there is one failing here, it is if the user did want to read from a credential
provider, the failure to read the credential will be lost, and the filesystem
will simply get the default value.
There is a side issue, that permission exceptions can surface as found not
found exceptions, which are then wrapped as generic IOEs in Configuration. It
will be hard and brittle to attempt to only respond to permission restrictions.
We could look at improving {{Configuration.getPassword()}} but that class is so
widely used, I am not in a rush to break things.
I think this means we have to add another option. Trying to be clever about
when to fall back versus when to rethrow the exception is doomed.
If this works for S3A, we will need to consider replicating it for ABFS.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
