[
https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904022#comment-15904022
]
Aaron Fabbri edited comment on HADOOP-14154 at 3/9/17 10:54 PM:
{quote}
Would that isAuthoritative flag have to be setup by higher level applications
like Pig/Hive/MR?
{quote}
No, it is internal to S3A. S3A can tell when it has the full listing for a
directory, and simply conveys that to the MetadataStore by setting the
isAuthoritative bit.
e.g. in {{S3Guard#dirListingUnion(..)}}, it always sets the flag when it puts
the listing into the MetadataStore, since this function always has the full
listing for the directory:
{code}
dirMeta.setAuthoritative(true); // This is the full directory contents
{code}
That codepath happens at the end of listStatus(), when it has finished
computing the full directory contents.
was (Author: fabbri):
{quote}
Would that isAuthoritative flag have to be setup by higher level applications
like Pig/Hive/MR?
{quote}
No, it is internal to S3A. S3A can tell when it has the full listing for a
directory, and simply conveys that to the MetadataStore by setting the
isAuthoritative bit.
e.g. in {{S3Guard#dirListingUnion(..)}}, it always sets the flag when it puts
the listing into the MetadataStore, since this function always has the full
listing for the directory:
{code}
dirMeta.setAuthoritative(true); // This is the full directory contents
{code}
> Set isAuthoritative flag when creating DirListingMetadata in DynamoDBMetaStore
> --
>
> Key: HADOOP-14154
> URL: https://issues.apache.org/jira/browse/HADOOP-14154
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HADOOP-14154-HADOOP-13345.001.patch,
> HADOOP-14154-HADOOP-13345.002.patch
>
>
> Currently {{DynamoDBMetaStore::listChildren}} does not populate
> {{isAuthoritative}} flag when creating {{DirListingMetadata}}.
> This causes additional S3 lookups even when users have enabled
> {{fs.s3a.metadatastore.authoritative}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
