[
https://issues.apache.org/jira/browse/HADOOP-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898693#comment-15898693
]
Aaron Fabbri commented on HADOOP-13914:
---------------------------------------
Thanks for the review [~liuml07]. Great comments..
{quote}
In DynamoDBMetadataStore#get(), we should check wantEmptyDirectoryFlag
{quote}
Good catch. Next patch will skip that extra work if the caller doesn't care.
I meant to do that but forgot.
{quote}
The isAuthoritative is false and we have children, we confidently trust the
child's existence though it's not authoritative (fully cached). Seems
reasonable.
{quote}
Yep.. And when we add support to DDB for tracking isAuthoritative, we may also
be able to answer true if {{authoritative && empty}}.
{quote}To simplify the if clause as if (!allowMissing()){quote}
That logic is intentional: {{allowMissing()}} means a MetadataStore may
discard contents. E.g. LocalMetadataStore discards entries based on LRU or
(eventually) TTL timeout. The case also covers NullMetadataStore which,
legally, discards everything.
Even if the MetadataStore is allowed to discard results, if it does return
results, we *still* run validation for the test to make sure the result is
correct. So, to summarize, allowMissing() means "test shouldn't fail if an
entry is missing" but otherwise, all validation should occur.
> s3guard: improve S3AFileStatus#isEmptyDirectory handling
> --------------------------------------------------------
>
> Key: HADOOP-13914
> URL: https://issues.apache.org/jira/browse/HADOOP-13914
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
> Attachments: HADOOP-13914-HADOOP-13345.000.patch,
> HADOOP-13914-HADOOP-13345.002.patch, HADOOP-13914-HADOOP-13345.003.patch,
> HADOOP-13914-HADOOP-13345.004.patch, HADOOP-13914-HADOOP-13345.005.patch,
> s3guard-empty-dirs.md, test-only-HADOOP-13914.patch
>
>
> As discussed in HADOOP-13449, proper support for the isEmptyDirectory() flag
> stored in S3AFileStatus is missing from DynamoDBMetadataStore.
> The approach taken by LocalMetadataStore is not suitable for the DynamoDB
> implementation, and also sacrifices good code separation to minimize
> S3AFileSystem changes pre-merge to trunk.
> I will attach a design doc that attempts to clearly explain the problem and
> preferred solution. I suggest we do this work after merging the HADOOP-13345
> branch to trunk, but am open to suggestions.
> I can also attach a patch of a integration test that exercises the missing
> case and demonstrates a failure with DynamoDBMetadataStore.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
