[
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665489#comment-15665489
]
Mingliang Liu edited comment on HADOOP-13449 at 11/15/16 12:22 AM:
-------------------------------------------------------------------
Thanks [~eddyxu] for your review and insightful comments.
# Yes we can define a new method for destroying in MS interface. At least, that
makes sense to DDB/MySQL stores. I'll upload a refined patch with this
addressed along with {{isAuthoritative}}.
# As to not creating the table in {{initialization}} or adding a flag
indicating the CREATE_IF_NOT_EXISTS , I did consider this, but was not sure
about this.
#- The 1st concern I have is about the mapping. I propose to map one
MetadataStore to one S3 bucket. MetadataStore does not have utility or general
functions for operating metadatas elsewhere except the specific S3 bucket (and
thus MetadataStore, say, a Table). After initialization all operations will
operate against the same S3 bucket; we don't need to specify the associated
Table for put/get/list etc operations; we don't need to specify the name for
creation and destroy either.
#- The second point is about the {{initialization()}} semantic. In
{{S3AFileSystem#initialize}}, it checks the bucket exists or not. It associate
the file system object with the bucket explicitly. If we initialize the
DDBMetadataStore from an S3AFileSystem, we associate the Table with the bucket
as well. Once initialized successfully, we know that the table is there, and
incoming operations are free to go. That's why I assumed we always create table
if not exists. If the table exists, the {{createTable()}} will use the existing
one instead of creating a new one. For destroy/deleteTable operations, the
target table being existent at initialization time is even more meaningful. The
overhead by now is considered small; we can of course optimize this part later.
Do this make sense?
was (Author: liuml07):
Thanks [~eddyxu] for your review and insightful comments.
# Yes we can define a new method for destroying in MS interface. At least, that
makes sense to DDB/MySQL stores. I'll upload a refined patch with this
addressed along with {{isAuthoritative}}.
# As to not creating the table in {{initialization}} or adding a flag
indicating the CREATE_IF_NOT_EXISTS , I did consider this, but was not sure
about this.
#- The 1st concern I have is about the mapping. I propose to map one
MetadataStore to one S3 bucket. MetadataStore does not have utility or general
functions for operating metadatas elsewhere except the specific S3 bucket (and
thus MetadataStore, say, a Table). After initialization all operations will
operate against the same S3 bucket; we don't need to specify the associated
Table for put/get/list etc operations; we don't need to specify the name for
creation and destroy either.
#- The second point is about the {{initialization()}} semantic. In
{{S3AFileSystem#initialize}}, it checks the bucket exists or not. It associate
the file system object with the bucket explicitly. If we initialize the
DDBMetadataStore from an S3AFileSystem, we associate the Table with the bucket
as well. Once initialized successfully, we know that the table is there, and
incoming operations are free to go. That's why I assumed we always create table
if not exists. If the table exists, the {{createTable()}} will use the existing
one instead of creating a new one. For destroy/deleteTable operations, the
target table being created/got at initialization time is even more meaningful.
The overhead by now is considered small; we can of course optimize this part
later. Do this make sense?
> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.000.patch,
> HADOOP-13449-HADOOP-13345.001.patch, HADOOP-13449-HADOOP-13345.002.patch,
> HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
