[
https://issues.apache.org/jira/browse/HADOOP-18435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599164#comment-17599164
]
Viraj Jasani commented on HADOOP-18435:
---------------------------------------
The way I came to know about this issue is by testing hbase on s3 (hbase WAL on
HDFS and actual HFiles on S3, accessed using S3A). I just wanted to see how
much perf improvement could be achieved by tuning fs.s3a.executor.capacity. I
tried with small bump, and then eventually very high bump but no significant
difference was observed. When I looked into the usages of
fs.s3a.executor.capacity, realized that it's not being used anywhere as per the
above comment.
Just to confirm that I have not missed any usecase, I provided value 0 to
fs.s3a.executor.capacity (adjusted min value from 1 to 0), and ran all S3A
tests with -Dscale, and not a single test failed. On the other hand, HBase
doesn't face any issues either.
> Remove usage of fs.s3a.executor.capacity
> ----------------------------------------
>
> Key: HADOOP-18435
> URL: https://issues.apache.org/jira/browse/HADOOP-18435
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
>
> When s3guard was part of s3a, DynamoDBMetadataStore was the only consumer of
> StoreContext that used throttled executor provided by StoreContext, which
> internally uses fs.s3a.executor.capacity to determine executor capacity for
> SemaphoredDelegatingExecutor. With the removal of s3guard from s3a, we should
> also remove fs.s3a.executor.capacity and it's usages as it's no longer being
> used by any StoreContext consumers. The config's existence and its
> description can be really confusing for the users.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
