[ https://issues.apache.org/jira/browse/HADOOP-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078883#comment-16078883 ]
Aaron Fabbri edited comment on HADOOP-14633 at 7/8/17 1:22 AM: --------------------------------------------------------------- Attaching v1 patch that gives almost 2x speedup for pathological create workload. - Adds new scale test for create performance, using a pathological case to stress S3guard ancestor creation. - Makes changes suggested in my comment linked in JIRA description. Performance tests in US West 2 from home laptop (decent latency to datacenter actually): 500 files of depth 10 via ITestS3ACreatePerformance {noformat} w/o s3guard, in *msec per create*: 521426 1.92 IOPS w/ s3guard, dynamo: 1153689 0.87 IOPS w/ s3guard, dynamo + this patch: 638544 1.56 IOPS {noformat} Currently running full suite of integration tests.. but wanted to post this before leaving for day.. Will followup with integration test results. was (Author: fabbri): Attaching v1 patch that gives almost 2x speedup for pathological create workload. - Adds new scale test for create performance, using a pathological case to stress S3guard ancestor creation. - Makes changes suggested in my comment linked in JIRA description. Performance tests in US West 2 from home laptop (low latency to datacenter actually): 500 files of depth 10 via ITestS3ACreatePerformance {noformat} w/o s3guard, in *msec per create*: 521426 1.92 IOPS - w/ s3guard, dynamo: 1153689 0.87 IOPS w/ s3guard, dynamo + this patch: 638544 1.56 IOPS {noformat} Currently running full suite of integration tests.. but wanted to post this before leaving for day.. Will followup with integration test results. > S3Guard: optimize create codepath > --------------------------------- > > Key: HADOOP-14633 > URL: https://issues.apache.org/jira/browse/HADOOP-14633 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Environment: > Reporter: Aaron Fabbri > Assignee: Aaron Fabbri > Priority: Minor > Fix For: HADOOP-13345 > > Attachments: HADOOP-14633-HADOOP-13345.001.patch > > > Following up on HADOOP-14457, a couple of things to do that will improve > create performance as I mentioned in the comment > [here|https://issues.apache.org/jira/browse/HADOOP-14457?focusedCommentId=16078465&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16078465] -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org