[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572676#comment-16572676 ]
Aaron Fabbri commented on HADOOP-14154: --------------------------------------- v3 patch looks good. I ran through all the integration tests with -Ddynamo -Ds3guard and they looked good. Running with -Dauth now. Would be good to have some sort of benchmark results that show a performance benefit. I tried using {{ITestS3ADirectoryPerformance}} but didn't see a big difference. This could be because any speedups in listStatus() are being lost when we write back the contents of the directory at the end of listStatus() in {{org.apache.hadoop.fs.s3a.s3guard.S3Guard#dirListingUnion}}: {noformat} if (changed && isAuthoritative) { dirMeta.setAuthoritative(true); // This is the full directory contents ms.put(dirMeta); } return dirMetaToStatuses(dirMeta); } {noformat} We could modify that test to do each treewalk multiple times, which would allow the caching (skipping S3 on listStatus()) to have a benefit, since each listing would happen more than once. Anyways, I think this is just about ready to be committed. Left to do are: - Documentation (mark as unsupported/experimental until we have better TTL expiry and more testing with it) - Nice to have: benchmarks showing a benefit (when I prototyped this before TCP-DS partitioned benchmark showed a good speedup, but it was a bit of a pain to set up). > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > ----------------------------------------------------------------------------- > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.0.0-beta1 > Reporter: Rajesh Balamohan > Assignee: Gabor Bota > Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org