[
https://issues.apache.org/jira/browse/HADOOP-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751069#comment-17751069
]
Steve Loughran commented on HADOOP-18838:
-----------------------------------------
# whatever is in core default takes priority, so don't worry about the code
# if you want to submit a PR fixing the rest, happy to accept it, though do
look at the "testing s3a" doc first. thanks
to work out what values a bucket has if not the source code defaults, the
"storediag" command in my cloudstore diagnostics library does this:
https://github.com/steveloughran/cloudstore
> Some fs.s3a.* config values are different in sources and documentation
> ----------------------------------------------------------------------
>
> Key: HADOOP-18838
> URL: https://issues.apache.org/jira/browse/HADOOP-18838
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.3.6
> Reporter: Maxim Martynov
> Priority: Major
>
> For config option {{fs.s3a.retry.throttle.interval}} default value in source
> code is {{500ms}}:
> {code:java}
> public static final String RETRY_THROTTLE_INTERVAL_DEFAULT = "500ms";
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L921
> In {{core-default.xml}} it has value {{100ms}}, but in the description
> {{500ms}}:
> {code:xml}
> <property>
> <name>fs.s3a.retry.throttle.interval</name>
> <value>100ms</value>
> <description>
> Initial between retry attempts on throttled requests, +/- 50%. chosen at
> random.
> i.e. for an intial value of 3000ms, the initial delay would be in the
> range 1500ms to 4500ms.
> Backoffs are exponential; again randomness is used to avoid the
> thundering heard problem.
> 500ms is the default value used by the AWS S3 Retry policy.
> </description>
> </property>
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1750
> This change introduced in HADOOP-16823.
> In Hadoop-AWS module documentation it has value {{1000ms}}:
> {code:xml}
> <property>
> <name>fs.s3a.retry.throttle.interval</name>
> <value>1000ms</value>
> <description>
> Interval between retry attempts on throttled requests.
> </description>
> </property>
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md?plain=1#L1223
> File was created in HADOOP-13786, and value is left unchanged since when.
> In performance tuning page it has up-to-date value {{500ms}}:
> {code:xml}
> <property>
> <name>fs.s3a.retry.throttle.interval</name>
> <value>500ms</value>
> <description>
> Interval between retry attempts on throttled requests.
> </description>
> </property>
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md?plain=1#L435
> This change introduced in HADOOP-15076.
> The same issue with:
> * {{fs.s3a.retry.throttle.limit}} - in source code it has value {{20}}, but
> in some documents still old value ${fs.s3a.attempts.maximum}
> * {{fs.s3a.connection.establish.timeout}} - in source code it has value
> {{50_000}}, in config file & documentation {{5_000}}
> * {{fs.s3a.attempts.maximum}} - in source code it has value {{10}}, in config
> file & documentation {{20}}
> * {{fs.s3a.threads.max}} - in source & documentation code it has value
> {{10}}, in config file {{64}}
> * {{fs.s3a.max.total.tasks}} - in source code & config it has value {{32}},
> in documentation {{5}}
> * {{fs.s3a.connection.maximum}} - in source code & config it has value
> {{96}}, in documentation {{15}} or {{30}}
> Please sync these values, outdated documentation is very painful to work with.
> As an idea, is it possible to use {{core-default.xml}} directly in
> documentation, or generate this documentation from docstrings in Java code?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
