[
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13449:
-----------------------------------
Attachment: HADOOP-13449-HADOOP-13345.001.patch
Thanks for asking [~eddyxu]. I attach the v1 patch for quick feedback.
# I changed the base unit test as the {{owner}}, {{group}} and {{permission}}
etc are not part of the metadata we're interested in by now.
# We store the {{is_empty}} for directory in the DynamoDB (DDB) metadata store
now. We have to update this information in a consistent and efficient way. We
don't want to check the parent directory every time we delete/put a file item.
At least we can optimize this when deleting a subtree.
# The contract assumes we create the direct parent directory (other ancestors
should be taken care of by the clients/callers) when putting a new file item. I
checked the in-memory local metadata store and it implements this idea. This
may be not efficient to DDB. Basically for putting X items, we have to issue
2X~3X DDB requests (X for putting file, X for checking its parent directories,
and possible X for updating its parent directories). I'm wondering if we can
also let the client/caller pre-create the direct parent directory as other
ancestors.
This is root cause of the only left 2 of 16 failing unit tests, i.e.
{{testPutDirListing}} and {{testPutNew}}.
# As to replacing FileStatus with S3AFileStatus in {{PathMetadata}}, I'm +0 for
the idea. If we do agree on the switch, [HADOOP-13736] is basically good to me.
If not, I can live with the similar way to {{S3AFileSystem}} vs. {{FileSystem}}
in the {{MetadataStore#initialize()}}.
# I need to review [HADOOP-13651] and revisit the patch after catching up the
current discussion. Will post v2 patch in one week. I will also handle the
{{isAuthoritative}} in the next patch. Storing an extra field is a good and
simple idea. Any idea how client sets/gets this value?
Thanks,
> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.000.patch,
> HADOOP-13449-HADOOP-13345.001.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
