[
https://issues.apache.org/jira/browse/HADOOP-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097552#comment-18097552
]
Steve Loughran commented on HADOOP-19937:
-----------------------------------------
* this really made the code internals a lot cleaner and removed some odd
failure modes, in case you are wondering why we didn't just leave it around;
* we don't need to to a head alongside a list as the list call will find any
marker, so a head is superfluous
be aware that on rename() calls, directory markers are *not* replicated, except
on empty dirs, which are required to preserve expectations of apps. We did
experiment once with dir recreation not happen, the way trino does, but s3a
apps are too broad and too many things break.
> When a folder is empty or renamed, fs.s3a.directory.marker.retention=delete
> should delete the parent markers
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-19937
> URL: https://issues.apache.org/jira/browse/HADOOP-19937
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.5.0, 3.4.1
> Reporter: Jayesh Lalwani
> Priority: Minor
>
> Right now, when a folder is deleted or renamed, S3A creates the parent
> markers for the empty folders. This makes sense when
> fs.s3a.directory.marker.retention is set to keep.
>
> However, when fs.s3a.directory.marker.retention is set to delete, it should
> skip creation of parent markers to maintain backward compatibility.
>
> {*}Code references (branch hadoop-3.4.2):{*}{*}{*}
>
> - S3AFileSystem.deleteWithoutCloseCheck() (line 4022) unconditionally calls
> maybeCreateFakeParentDirectory(path) after delete
> - OperationCallbacksImpl.finishRename() (line 3091) unconditionally calls
> maybeCreateFakeParentDirectory(sourceRenamed) after rename
> - Neither call checks keepDirectoryMarkers() before creating the marker
>
> {*}Impact:{*}{*}{*}
>
> Customers migrating from EMRFS to S3A (e.g., EMR 7.8 → 7.10) see orphan
> zero-byte directory markers accumulating after FileSystem.rename()
> operations. These markers cause applications (e.g., Spark Structured
> Streaming) to issue nnecessary S3 LIST calls on empty paths, increasing S3
> costs. Setting fs.s3a.directory.marker.retention=delete does not prevent this
> behavior.
>
> {*}Proposed fix:{*}{*}{*}
>
> Guard the maybeCreateFakeParentDirectory() calls with the marker retention
> policy:
>
> {{// In finishRename():}}
> {{if (keepDirectoryMarkers(sourceRenamed.getParent())) {}}
> {{ maybeCreateFakeParentDirectory(sourceRenamed);}}
> {{ }}}
>
> {{// In deleteWithoutCloseCheck():}}
> {{if (keepDirectoryMarkers(path.getParent())) {}}
> {{ maybeCreateFakeParentDirectory(path);}}
> {{ }}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
