[
https://issues.apache.org/jira/browse/HADOOP-18435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611287#comment-17611287
]
Viraj Jasani commented on HADOOP-18435:
---------------------------------------
I agree that bounded threadpool is used by more usecases now, and also that we
do need an executor in store context. But we already have the consumers of
{code:java}
public ExecutorService createThrottledExecutor(int capacity) {
return new SemaphoredDelegatingExecutor(executor,
capacity, true);
} {code}
We don't have any consumers of the StoreContext#executorCapacity and hence
*_createThrottledExecutor()_* is of no use, only *_createThrottledExecutor(int
capacity)_* is being used.
Hence, when we say that the config _fs.s3a.executor.capacity_ is used to
represent the capacity of the executor queues other than block upload, the
users would tend to use the config to tune the executor queues but would not
see any difference in the use-case behaviors.
The use-cases like prefetch, vectored IO, huge file write with
S3ABlockOutputStream etc do use SemaphoredDelegatingExecutor with bounded
thread pool, however none of them are using _fs.s3a.executor.capacity_ to
determine the bounded thread pool capacity.
> Remove usage of fs.s3a.executor.capacity
> ----------------------------------------
>
> Key: HADOOP-18435
> URL: https://issues.apache.org/jira/browse/HADOOP-18435
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
>
> When s3guard was part of s3a, DynamoDBMetadataStore was the only consumer of
> StoreContext that used throttled executor provided by StoreContext, which
> internally uses fs.s3a.executor.capacity to determine executor capacity for
> SemaphoredDelegatingExecutor. With the removal of s3guard from s3a, we should
> also remove fs.s3a.executor.capacity and it's usages as it's no longer being
> used by any StoreContext consumers. The config's existence and its
> description can be really confusing for the users.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
