Lei (Eddy) Xu commented on HADOOP-13449:

Thanks for posting the patch,[~liuml07]. 

The concept looks very reasonable in general. And I like the schema. I 
understand that this is a WIP patch. So I'd list the following suggestions for 

* Using local dynamodb local mode in test is really nice.
* We should store more metadata beside {{is_directory}} in metadata. so that we 
can reconstruct {{S3AFileStatus}} in {{itemToPathMetadata()}}.  Especially, do 
you think it is worth to store {{S3AFileStatus#isEmptyDirectory}} as well?
* To this extend, I think {{PathMetadata}} should take {{S3AFileStatus()}} 
instead of {{FileStatus}}. So that S3 specific attributes are more easily to be 
* {{DynamoDBMetadataStore}} should have a {{deleteTable}} on par with 
{{initTable}}. Both {{initTable}} and {{deleteTable}} should be package-wide 
visible so that it can be used from CLI tools. 
try {
} catch (InterruptedException e) {
   LOG.warn("Interrupted while waiting for DynamoDB table {} active",
      tableName, e);

Should it throw {{IOE}} to indicate the failure?

* When do {{table.query()}} in {{listChildren()}},  the query might return 
partial results because the returned dataset is large. You can use {{ 
QueryResult#LastEvaluatedKey()}} for the following calls.

* DynamoDB {{tableName}} should be able to be specified in configuration, i.e., 
considering that multiple ETL jobs might running against the same dataset with 
different purposes and different lifetimes, using different tables could allow 
such jobs managed the lifetime of dynamodb tables by themself. 

Thanks for the nice work!

> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>                 Key: HADOOP-13449
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13449
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Mingliang Liu
>         Attachments: HADOOP-13449-HADOOP-13345.wip.patch
> Provide an implementation of the metadata store backed by DynamoDB.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to