[
https://issues.apache.org/jira/browse/HADOOP-14226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946121#comment-15946121
]
Aaron Fabbri commented on HADOOP-14226:
---------------------------------------
Hi [~liuml07]. I was thinking about this issue, and it seems that
DyamoDBMetadataStore should make sure that is has entries for the ancestors of
any path. This is an implementation detail of that particular MetadataStore.
I was thinking that {{DynamoDBMetadataStore#move()}} needs to do the same thing
that {{put()}} does: use a function like
{{DynamoDBMetadataStore#fullPathsToPut()}} to make sure it has records for each
directory up to the root.
This is different than HADOOP-14236, where S3A FS was failing to be consistent
in the way it is calling the MetadataStore. In that bug, the fix only supplies
intermediate paths that are "inferred" in S3. It correctly does not continue
adding paths all the way up to the root. So I think move() also needs to add
the directory entries from the root down to the directory being moved.
> S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
> ---------------------------------------------------------------------
>
> Key: HADOOP-14226
> URL: https://issues.apache.org/jira/browse/HADOOP-14226
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Mingliang Liu
> Assignee: Mingliang Liu
> Priority: Minor
> Attachments: HADOOP-14226-HADOOP-13345.000.patch
>
>
> After running {{ITestDynamoDBMetadataStoreScale}}, the test data is not
> cleaned up. There is a call to {{clearMetadataStore(ms, count);}} in the
> finally clause though. The reason is that, the internally called method
> {{DynamoDBMetadataStore::deleteSubtree()}} is assuming there should be an
> item for the parent dest path:
> {code}
> parent=/fake-bucket, child=moved-here, is_dir=true
> {code}
> In DynamoDBMetadataStore implementation, we assume that _if a path exists,
> all its ancestors will also exist in the table_. We need to pre-create dest
> path to maintain this invariant so that test data can be cleaned up
> successfully.
> I think there may be other tests with the same problem. Let's
> identify/address them separately.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
