liuzix opened a new pull request, #29598:
URL: https://github.com/apache/flink/pull/29598
## What is the purpose of the change
`NativeS3FileSystem.getFileStatus` treats every successful HeadObject whose
Content-Length is 0 (or absent) as a directory marker: it lists the prefix `key
+ "/"` and, when nothing is under it, throws `FileNotFoundException`. An empty
object that is a plain file (a key without a trailing slash and without
children) is therefore reported as not found: `exists()` returns false for it,
`open()` and `delete()` treat it as absent, and `create()` with
`WriteMode.NO_OVERWRITE` overwrites it instead of refusing. The Hadoop S3A and
Presto file systems report such an object as a file of length 0, so a job
behaves differently on `flink-s3-fs-native` than on the other two. One real
case: an application that writes an empty "startup marker" object on its first
start and checks `exists()` on later starts never saw its marker on the native
file system and re-ran its first-start path every time.
This pull request makes `getFileStatus` report a zero-length object as a
file unless its key is a directory marker (ends in `/`) or has children under
`key + "/"`.
Jira: FLINK-XXXXX (the issue is being filed; the title and commit message
get the id once it exists).
## Brief change log
- `getFileStatus` returns a directory for a key ending in `/`; for a
zero-length object it returns a directory only when children exist under `key +
"/"` (one `ListObjectsV2` with `max-keys=1`) and a file of length 0 otherwise;
a missing key keeps the existing behavior (a directory when children exist,
`FileNotFoundException` otherwise).
- The children listing moved into a private `hasChildren`, shared with
`getDirectoryStatus`, and reads `commonPrefixes().isEmpty()` instead of
`hasCommonPrefixes()`, which is true for an explicitly set empty list.
- New `NativeS3FileSystemTest` with ten unit tests against an in-process
HTTP stub standing in for S3 (the JDK `HttpServer` behind the real
`S3ClientProvider` client, since the module's checkstyle forbids Mockito).
## Verifying this change
This change added tests and can be verified as follows:
- `NativeS3FileSystemTest` covers: the empty object without children is a
file of length 0 with the HEAD's `Last-Modified`, `exists()` is true and
exactly one listing with `prefix=key/` and `max-keys=1` is made; the empty
object with children is a directory; a key ending in `/` is a directory with
and without content, with no listing; a non-empty object is a file of its
length; a missing key with children, or with a common prefix, is a directory; a
missing key without children throws `FileNotFoundException` and `exists()` is
false; `create(path, NO_OVERWRITE)` over an empty object throws "File already
exists".
- The module's unit tests (`mvn -pl flink-filesystems/flink-s3-fs-native
test`, 334 tests, 10 of them new) passed on the author's tree, whose copies of
`NativeS3FileSystem`, `S3ClientProvider`, `S3FileStatus` and `S3UriUtils` are
identical to master's; this pull request's CI runs them on master.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: don't know (the file
system's `exists()` answer changes for empty objects)
- The S3 file system connector: yes
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code 2.1.310, model claude-fable-5-1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
