[
https://issues.apache.org/jira/browse/HADOOP-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16919525#comment-16919525
]
Arun Ravi M V edited comment on HADOOP-16540 at 8/30/19 1:11 PM:
-----------------------------------------------------------------
Yes, you are right, the main reason for this ticket is user credentials. I have
a situation where a large number of datasets (a few thousand) is located at a
single s3 bucket. I am trying to introduce Role-based access control here, AWS
policies have size limitations and cannot be used as the only solution. In this
case, I would like to define caching per dataset (bucket + root s3 prefix)
instead of doing it at the s3 bucket level.
was (Author: arunravimv):
Yes, you are right, the main reason for this ticket is user credentials. I have
a situation where a large number of datasets (a few thousand) is located at a
single s3 bucket. I am trying to introduce Role-based access control here, AWS
policies have size limitations and cannot be used as the only solution. In this
case, I would like to define caching per dataset (bucket + root s3 prefix)
instead of doing it at the s3 bucket level.
> Pluggable Filesystem Caching Support in FileSystem Class
> --------------------------------------------------------
>
> Key: HADOOP-16540
> URL: https://issues.apache.org/jira/browse/HADOOP-16540
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Affects Versions: 3.3.0
> Reporter: Arun Ravi M V
> Priority: Major
>
> Provide an option to use Custom Cache Class in FileSystem Class. Currently,
> the caching is enabled by default and uses the URI schema and authority value
> to determine whether to create a new FS instance for the given URI or to
> fetch an already existing one from the cache.
> In case of AWS S3 FS Impl, for an S3 path, the authority name will be bucket
> name, ie Filesystem object will be cached at the bucket level, but providing
> a custom caching logic can empower the user to cache it at some prefix level
> and provide more flexibility.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
