[
https://issues.apache.org/jira/browse/HADOOP-18435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611281#comment-17611281
]
Viraj Jasani commented on HADOOP-18435:
---------------------------------------
[~mthakur] [[email protected]] even after HADOOP-18347 is merged, i still see
that config *_fs.s3a.executor.capacity_* is not being used anywhere and merely
creating confusion for users.
We can apply this patch and run tests with scale profile to confirm the same:
{code:java}
diff --git
a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
index 17cd228dc1b..de7a9b84759 100644
--- a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
+++ b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
@@ -1523,7 +1523,7 @@
<property>
<name>fs.s3a.executor.capacity</name>
- <value>16</value>
+ <value>0</value>
<description>The maximum number of submitted tasks which is a single
operation (e.g. rename(), delete()) may submit simultaneously for
execution -excluding the IO-heavy block uploads, whose capacity
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
index 856d1dfb97b..807ba23913d 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
@@ -417,7 +417,7 @@ private Constants() {
* upload, where {@link #FAST_UPLOAD_ACTIVE_BLOCKS} is used instead.
* Value: {@value}
*/
- public static final int DEFAULT_EXECUTOR_CAPACITY = 16;
+ public static final int DEFAULT_EXECUTOR_CAPACITY = 0;
// Private | PublicRead | PublicReadWrite | AuthenticatedRead |
// LogDeliveryWrite | BucketOwnerRead | BucketOwnerFullControl
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
index 3e6f2322d3b..ca4abaf0850 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
@@ -783,7 +783,7 @@ private void initThreadPools(Configuration conf) {
name + "-unbounded"));
unboundedThreadPool.allowCoreThreadTimeOut(true);
executorCapacity = intOption(conf,
- EXECUTOR_CAPACITY, DEFAULT_EXECUTOR_CAPACITY, 1);
+ EXECUTOR_CAPACITY, DEFAULT_EXECUTOR_CAPACITY, 0);
if (prefetchEnabled) {
final S3AInputStreamStatistics s3AInputStreamStatistics =
statisticsContext.newInputStreamStatistics();
{code}
The reason why this patch would have no impact is because as I mentioned in the
above comment, this config is only used by:
{code:java}
/**
* Create a new executor with the capacity defined in
* {@link #executorCapacity}.
* @return a new executor for exclusive use by the caller.
*/
public ExecutorService createThrottledExecutor() {
return createThrottledExecutor(executorCapacity);
}{code}
The executorCapacity here is derived from the config
{*}_fs.s3a.executor.capacity_{*}.
Hence, even after merging HADOOP-18347, the fact remains the same that when
s3guard was part of s3a, DynamoDBMetadataStore was the only consumer of the
above method (i.e. createThrottledExecutor()). Now that s3guard is no longer
present, this config's presence is just creating more confusions.
> Remove usage of fs.s3a.executor.capacity
> ----------------------------------------
>
> Key: HADOOP-18435
> URL: https://issues.apache.org/jira/browse/HADOOP-18435
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
>
> When s3guard was part of s3a, DynamoDBMetadataStore was the only consumer of
> StoreContext that used throttled executor provided by StoreContext, which
> internally uses fs.s3a.executor.capacity to determine executor capacity for
> SemaphoredDelegatingExecutor. With the removal of s3guard from s3a, we should
> also remove fs.s3a.executor.capacity and it's usages as it's no longer being
> used by any StoreContext consumers. The config's existence and its
> description can be really confusing for the users.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
