[
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593462#comment-15593462
]
Aaron Fabbri edited comment on HADOOP-13449 at 10/21/16 12:21 AM:
------------------------------------------------------------------
Thanks for feedback [~eddyxu]. Turns out a user can provide URIs for any number
of buckets. They will each have their own S3AFileSystem of course.
I actually ran into this running integration tests (this is the last failure I
have for my S3AFileSystem integration work). Some of the S3A Scale integration
tests hit s3a://landsat-pds in addition to the test filesystem. This caused a
failure when I had paths from both buckets in my MetadataStore, without the
leading scheme+host (bucket) to differentiate between the two.
I'm proposing that any FS client that supplies a Path without a host (bucket)
component will get an exception, if it should have one (i.e. S3AFileSystem
should, RawLocalFilesystem should not). I'll call it out when I post my next
patch on HADOOP-13651.
was (Author: fabbri):
Thanks for feedback [~eddyxu]. Turns out a user can provide URIs for any number
of buckets. They will each have their own S3AFileSystem of course.
I actually ran into this running integration tests (this is the last failure I
have for my S3AFileSystem integration work). Some of the S3A Scale integration
tests hit s3a://landsat-pds in addition to the test filesystem. This caused a
failure when I had paths from both buckets in my MetadataStore, without the
leading scheme+host (bucket) to differentiate between the two.
I'm proposing that any FS client that supplies a Path without a host component
will get an exception, if it should have one (i.e. S3AFileSystem should,
RawLocalFilesystem should not). I'll call it out when I post my next patch on
HADOOP-13651.
> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.wip.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
