[
https://issues.apache.org/jira/browse/HADOOP-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625235#comment-15625235
]
Steve Loughran commented on HADOOP-13651:
-----------------------------------------
# I've actually been talking with [~rajesh.balamohan] about pulling that
initial bucket check (HADOOP-13379) . It adds measurable delays to all FS
instance creation —and permission errors will show up later on anyway. The
tricky bit is having a later 40x failure be uprated to a "there is no such
bucket" rather than "you can't access a file". You can save 500+mS by removing
an otherwise needless HTTP request; sometimes it can even take longer. I think
I would like to cut it, if the failure can be graceful (some tracking of if a
request has ever succeeded, on first auth failure, go from simple translation
to adding "check bucket exists"/actually falling back to a second check)
# if a user has valid read credentials, bucket exist check (currently) fails in
init. IF this is delayed, then the first s3 read/write will fail
# bucket nonexistent -> 404? 410? -> FNFE
# bucket exists but caller not authed -> 401? 403? -> AccessDeniedException
# if a user has read but not write credentials, any attempt to do multipart
purge will fail; that's now caught & downgraded.
# if a user has no credentials, then, if the auth chain has to be set up to
allow anonymous access, then they'll try an anonymous auth (not a default
option), and they'll get read access to any bucket declared public readable.
> S3Guard: S3AFileSystem Integration with MetadataStore
> -----------------------------------------------------
>
> Key: HADOOP-13651
> URL: https://issues.apache.org/jira/browse/HADOOP-13651
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
> Attachments: HADOOP-13651-HADOOP-13345.001.patch,
> HADOOP-13651-HADOOP-13345.002.patch, HADOOP-13651-HADOOP-13345.003.patch
>
>
> Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata
> consistency and caching.
> Implementation should have minimal overhead when no MetadataStore is
> configured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
