Jayesh Lalwani created HADOOP-19937:
---------------------------------------
Summary: When a folder is empty or renamed,
fs.s3a.directory.marker.retention=delete should delete the parent markers
Key: HADOOP-19937
URL: https://issues.apache.org/jira/browse/HADOOP-19937
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.4.1, 3.5.0
Reporter: Jayesh Lalwani
Right now, when a folder is deleted or renamed, S3A creates the parent markers
for the empty folders. This makes sense when fs.s3a.directory.marker.retention
is set to keep.
However, when fs.s3a.directory.marker.retention is set to delete, it should
skip creation of parent markers to maintain backward compatibility.
{*}Code references (branch hadoop-3.4.2):{*}{*}{*}
- S3AFileSystem.deleteWithoutCloseCheck() (line 4022) unconditionally calls
maybeCreateFakeParentDirectory(path) after delete
- OperationCallbacksImpl.finishRename() (line 3091) unconditionally calls
maybeCreateFakeParentDirectory(sourceRenamed) after rename
- Neither call checks keepDirectoryMarkers() before creating the marker
{*}Impact:{*}{*}{*}
Customers migrating from EMRFS to S3A (e.g., EMR 7.8 → 7.10) see orphan
zero-byte directory markers accumulating after FileSystem.rename() operations.
These markers cause applications (e.g., Spark Structured Streaming) to issue
nnecessary S3 LIST calls on empty paths, increasing S3 costs. Setting
fs.s3a.directory.marker.retention=delete does not prevent this behavior.
{*}Proposed fix:{*}{*}{*}
Guard the maybeCreateFakeParentDirectory() calls with the marker retention
policy:
{{// In finishRename():}}
{{if (keepDirectoryMarkers(sourceRenamed.getParent())) {}}
{{ maybeCreateFakeParentDirectory(sourceRenamed);}}
{{ }}}
{{// In deleteWithoutCloseCheck():}}
{{if (keepDirectoryMarkers(path.getParent())) {}}
{{ maybeCreateFakeParentDirectory(path);}}
{{ }}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
