[
https://issues.apache.org/jira/browse/HADOOP-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078883#comment-16078883
]
Aaron Fabbri edited comment on HADOOP-14633 at 7/8/17 1:22 AM:
---------------------------------------------------------------
Attaching v1 patch that gives almost 2x speedup for pathological create
workload.
- Adds new scale test for create performance, using a pathological case to
stress S3guard ancestor creation.
- Makes changes suggested in my comment linked in JIRA description.
Performance tests in US West 2 from home laptop (decent latency to datacenter
actually):
500 files of depth 10 via ITestS3ACreatePerformance
{noformat}
w/o s3guard, in *msec per create*:
521426
1.92 IOPS
w/ s3guard, dynamo:
1153689
0.87 IOPS
w/ s3guard, dynamo + this patch:
638544
1.56 IOPS
{noformat}
Currently running full suite of integration tests.. but wanted to post this
before leaving for day.. Will followup with integration test results.
was (Author: fabbri):
Attaching v1 patch that gives almost 2x speedup for pathological create
workload.
- Adds new scale test for create performance, using a pathological case to
stress S3guard ancestor creation.
- Makes changes suggested in my comment linked in JIRA description.
Performance tests in US West 2 from home laptop (low latency to datacenter
actually):
500 files of depth 10 via ITestS3ACreatePerformance
{noformat}
w/o s3guard, in *msec per create*:
521426
1.92 IOPS
- w/ s3guard, dynamo:
1153689
0.87 IOPS
w/ s3guard, dynamo + this patch:
638544
1.56 IOPS
{noformat}
Currently running full suite of integration tests.. but wanted to post this
before leaving for day.. Will followup with integration test results.
> S3Guard: optimize create codepath
> ---------------------------------
>
> Key: HADOOP-14633
> URL: https://issues.apache.org/jira/browse/HADOOP-14633
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Environment:
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
> Priority: Minor
> Fix For: HADOOP-13345
>
> Attachments: HADOOP-14633-HADOOP-13345.001.patch
>
>
> Following up on HADOOP-14457, a couple of things to do that will improve
> create performance as I mentioned in the comment
> [here|https://issues.apache.org/jira/browse/HADOOP-14457?focusedCommentId=16078465&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16078465]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
