[
https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572676#comment-16572676
]
Aaron Fabbri commented on HADOOP-14154:
---------------------------------------
v3 patch looks good. I ran through all the integration tests with -Ddynamo
-Ds3guard and they looked good. Running with -Dauth now.
Would be good to have some sort of benchmark results that show a performance
benefit. I tried using {{ITestS3ADirectoryPerformance}} but didn't see a big
difference. This could be because any speedups in listStatus() are being lost
when we write back the contents of the directory at the end of listStatus() in
{{org.apache.hadoop.fs.s3a.s3guard.S3Guard#dirListingUnion}}:
{noformat}
if (changed && isAuthoritative) {
dirMeta.setAuthoritative(true); // This is the full directory contents
ms.put(dirMeta);
}
return dirMetaToStatuses(dirMeta);
}
{noformat}
We could modify that test to do each treewalk multiple times, which would allow
the caching (skipping S3 on listStatus()) to have a benefit, since each listing
would happen more than once.
Anyways, I think this is just about ready to be committed. Left to do are:
- Documentation (mark as unsupported/experimental until we have better TTL
expiry and more testing with it)
- Nice to have: benchmarks showing a benefit (when I prototyped this before
TCP-DS partitioned benchmark showed a good speedup, but it was a bit of a pain
to set up).
> Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
> -----------------------------------------------------------------------------
>
> Key: HADOOP-14154
> URL: https://issues.apache.org/jira/browse/HADOOP-14154
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0-beta1
> Reporter: Rajesh Balamohan
> Assignee: Gabor Bota
> Priority: Minor
> Attachments: HADOOP-14154-HADOOP-13345.001.patch,
> HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf,
> HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch,
> HADOOP-14154.003.patch
>
>
> Add support for "authoritative mode" for DynamoDBMetadataStore.
> The missing feature is to persist the bit set in
> {{DirListingMetadata.isAuthoritative}}.
> This topic has been super confusing for folks so I will also file a
> documentation Jira to explain the design better.
> We may want to also rename the DirListingMetadata.isAuthoritative field to
> .isFullListing to eliminate the multiple uses and meanings of the word
> "authoritative".
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
