dannycjones commented on code in PR #5689: URL: https://github.com/apache/hadoop/pull/5689#discussion_r1214190161
########## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/directory_markers.md: ########## @@ -12,35 +12,40 @@ limitations under the License. See accompanying LICENSE file. --> -# Experimental: Controlling the S3A Directory Marker Behavior +# Controlling the S3A Directory Marker Behavior -This document discusses an experimental feature of the S3A -connector since Hadoop 3.3.1: the ability to retain directory -marker objects above paths containing files or subdirectories. +This document discusses an performance feature of the S3A +connector: directory markers are not deleted unless the +client is explicitly configured to do so. ## <a name="compatibility"></a> Critical: this is not backwards compatible! This document shows how the performance of S3 I/O, especially applications creating many files (for example Apache Hive) or working with versioned S3 buckets can increase performance by changing the S3A directory marker retention policy. -Changing the policy from the default value, `"delete"` _is not backwards compatible_. +The default policy in this release of hadoop is "keep", +which _is not backwards compatible_ with hadoop versions +released before 2021. -Versions of Hadoop which are incompatible with other marker retention policies, -as of August 2020. +The compatibility table of older releases is as follows: -| Branch | Compatible Since | Supported | -|------------|------------------|---------------------| -| Hadoop 2.x | n/a | WONTFIX | -| Hadoop 3.0 | check | Read-only | -| Hadoop 3.1 | check | Read-only | -| Hadoop 3.2 | check | Read-only | -| Hadoop 3.3 | 3.3.1 | Done | +| Branch | Compatible Since | Supported | Released | +|------------|------------------|-----------|----------| +| Hadoop 2.x | 2.10.2 | Read-only | 05/2022 | +| Hadoop 3.0 | n/a | WONTFIX | | +| Hadoop 3.1 | n/a | WONTFIX | | +| Hadoop 3.2 | 3.2.2 | Read-only | 01/2022 | +| Hadoop 3.3 | 3.3.1 | Done | 01/2021 | Review Comment: Thanks for updating this with the extra info. Do we know why the Hadoop webpages aren't formatting the original table? https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/directory_markers.html#The_Problem_with_Directory_Markers -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
