[
https://issues.apache.org/jira/browse/HADOOP-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017948#comment-17017948
]
Steve Loughran edited comment on HADOOP-16807 at 1/17/20 12:16 PM:
-------------------------------------------------------------------
Something like this has surfaced before, HADOOP-16540, and closed as a wontfix.
I don't see how your proposal will address the fundamental problems there
-specifically the assumption that rename() only need one set of credentials to
work across the store, and that therefore operations like distCP can acquire a
single FS instance and use it for all this work,
Again, S3 seems to be the goal. The way I would envisage this being fixed in
S3A is via HADOOP-16134 and dynamically creating/binding to a credential
provider unique to the FS operation being performed. We'd create this at the
start of the action (create, open, rename) and propagate it through all
subsequent interactions with the store. That way: one FS, multiple credential
providers.
I don't see why we need to make fundamental changes in the FS cache given that
such a design appears to be better due to the ability to dynamically create an
authentication chain rather have a large (expensive) pool of FS instances with
their auth chains created upfront.
I'm intermittently active on that context, most recently on tuesday. Look on
https://github.com/steveloughran/hadoop for branches with {{s3/HADOOP-16134}}
as the prefix.
was (Author: [email protected]):
Something like this has surfaced before, HADOOP-16540, and closed as a wontfix.
I don't see how your proposal will address the fundamental problems there
-specifically the assumption that rename() only need one set of credentials to
work across the store, and that therefore operations like distCP can acquire a
single FS instance and use it for all this work,
Again, S3 seems to be the goal. The way I would envisage this being fixed in
S3A is via HADOOP-16134 and dynamically creating/binding to a credential
provider unique to the FS operation being performed. We'd create this at the
start of the action (create, open, rename) and propagate it through all
subsequent interactions with the store. That way: one FS, multiple credential
providers.
I don't see why we need to make fundamental changes in the FS cache given that
such a design appears to be better due to the ability to dynamically create an
authentication chain rather have a large (expensive) pool of FS instances with
their auth chains created upfront.
I'm intermittently active on that context, most recently on tuesday. Look on
https://github.com/steveloughran/hadoop for branches with s3/HADOOP-16134 as
the prefix.
> Enable Filesystem caching to optionally include URI Path
> --------------------------------------------------------
>
> Key: HADOOP-16807
> URL: https://issues.apache.org/jira/browse/HADOOP-16807
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Reporter: David Dudley
> Priority: Major
>
> Implementing AWSCredentialsProviders that dynamically retrieve STS tokens
> based on the URI being accessed fail if Filesystem caching is enabled and the
> job accesses more than one URI Path within the same bucket.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
