[
https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562984#comment-16562984
]
Aaron Fabbri edited comment on HADOOP-14154 at 7/31/18 5:46 AM:
----------------------------------------------------------------
Thanks for the patch
edit: I completely read the wrong patch.. ignore previous comment. Will
re-review in the morning.
was (Author: fabbri):
Thanks for the patch.
{quote}
* Added {{PathMetadata#isAuthoritativeDir}} because {{DynamoDBMetadataStore}}
stores {{PathMetadata}} in ddb, so that was a logical choice.
{quote}
I did not see this in your diff, it may be missing some files?
This may be a quick way to get the code working because PathMetadata are
translated DDB rows, but I think it confuses the MetadataStore API. I think
the API should probably remain as is (you set Authoritative bit on directories,
not files). This probably means you need to change the
PathMetadataDynamoDBTranslation logic a bit, though.
Also the testAuthoritative case you added; should that go in the base class
instead of the TestDynamoDBMetadataStore subclass? I think it should. Any
MetadataStore should behave the same if it supports persisting the auth bit
(they all will now, except for Null of course). There is already some code
there, though. See testListChildrenAuthoritative()
{noformat}
---
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java
+++
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java
@@ -188,6 +188,7 @@
private String tableName;
private Configuration conf;
private String username;
+ private boolean allowAuthoritative;
{noformat}
The MetadataStore just persists the auth bit on directories. The FS (S3A) is
what decides which listing is a full/complete/authoritative listing. So, I
don't think you need this here.
{noformat}
@@ -365,7 +374,7 @@ public DirListingMetadata listChildren(Path path) throws
IOException {
return (metas.isEmpty() && get(path) == null)
? null
- : new DirListingMetadata(path, metas, false);
+ : new DirListingMetadata(path, metas, allowAuthoritative);
} catch (AmazonClientException e) {
throw translateException("listChildren", path, e);
{noformat}
A DirListingMetadata is authoritative=true iff (if and only if) the FS set it
when it previously called put(). Returning true based on the config is not
correct here.
That config option "allow authoritative" should only be visible in the FS (S3A)
code; it declares that S3A *may* treat MS results as authoritative if the bit
is set. The MetadataStore's job is simply to return (persist) what it was told
earlier by the FS when it returns a DirListingMetadata.
So if MS.put(/some/dir, auth=true)
then MS.get(/some/dir) can return with auth=true (assuming a prune() or
something else has not invalidated the listing for that dir)
but if MS.put(/some/dir, auth=false)
then MS.get(/some/dir) must return with auth=false.
This behavior is completely independent from the value of
fs.s3a.metadatastore.authoritative (that is only used by the FS to skip round
trips to S3 for listings)
Thanks for digging into this feature--I know it is confusing but I hope this
helps some.
> Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
> -----------------------------------------------------------------------------
>
> Key: HADOOP-14154
> URL: https://issues.apache.org/jira/browse/HADOOP-14154
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0-beta1
> Reporter: Rajesh Balamohan
> Assignee: Gabor Bota
> Priority: Minor
> Attachments: HADOOP-14154-HADOOP-13345.001.patch,
> HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf,
> HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch
>
>
> Add support for "authoritative mode" for DynamoDBMetadataStore.
> The missing feature is to persist the bit set in
> {{DirListingMetadata.isAuthoritative}}.
> This topic has been super confusing for folks so I will also file a
> documentation Jira to explain the design better.
> We may want to also rename the DirListingMetadata.isAuthoritative field to
> .isFullListing to eliminate the multiple uses and meanings of the word
> "authoritative".
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
