Steve Loughran created HADOOP-16567:

             Summary: S3A Secret access to fall back to XML if credential 
provider raises IOE.
                 Key: HADOOP-16567
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: 3.1.2
            Reporter: Steve Loughran

This is hive related. Hive can put secrets into a JCEKS file which only hive 
may read.

S3A file systems created on behalf of a user do not have access to this file. 
Yet it is listed as the credential provider in the hadoop.credential.providers 
option in core-site -and that is marked as final. When the S3 a initializre() 
method looks up passwords and encryption keys, the failure to open the file 
raises an IOE -and the FS cannot be instantiated.

Proposed: {{S3AUtils.lookupPassword()}} to catch such exceptions, and fall back 
to using {{Configuration.get()}} and so retrieve any property in the XML. If 
there is one failing here, it is if the user did want to read from a credential 
provider, the failure to read the credential will be lost, and the filesystem 
will simply get the default value.

There is a side issue, that permission exceptions can surface as found not 
found exceptions, which are then wrapped as generic IOEs in Configuration. It 
will be hard and brittle to attempt to only respond to permission restrictions. 
We could look at improving {{Configuration.getPassword()}} but that class is so 
widely used, I am not in a rush to break things.

I think this means we have to add another option. Trying to be clever about 
when to fall back versus when to rethrow the exception is doomed.

If this works for S3A, we will need to consider replicating it for ABFS. 

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to