eskabetxe opened a new pull request, #29420: URL: https://github.com/apache/flink/pull/29420
## What is the purpose of the change Fixes [FLINK-40942](https://issues.apache.org/jira/browse/FLINK-40942): in `flink-s3-fs-native`, `getFileStatus()` / `exists()` on a bucket-root path (`s3://bucket` or `s3://bucket/`) threw `SdkClientException: Key cannot be empty`, because `extractKey()` yields an empty key and the AWS SDK v2 `HeadObject` marshaller rejects it client-side. This is a regression for users migrating from `flink-s3-fs-hadoop`, where the bucket root is an always-existing directory (e.g. Paimon's `HiveCatalog` calls `exists(warehouse)` at catalog creation, and the docs show checkpoint dirs of the form `s3://<bucket>/`). ## Brief change log - `NativeS3FileSystem#getFileStatus`: empty-key (bucket-root) paths now skip `HeadObject`, verify the bucket via `HeadBucket` and return a directory status, matching S3A semantics. `NoSuchBucketException` (and bare 404s from S3-compatible stores) map to `FileNotFoundException` so `exists()` returns `false`; other S3 errors (e.g. 403) still surface as `IOException`. - Since root paths now resolve, guard the newly reachable operations against silent data loss: - `delete(bucket-root, *)` throws `IOException` instead of resolving to a directory whose recursive delete silently no-ops and returns `true` (S3A also refuses root deletes). - `open()` on a directory (including the bucket root) throws `FileNotFoundException` instead of returning an empty stream. ## Verifying this change This change added tests and can be verified as follows: - New `NativeS3FileSystemITCase` cases (SeaweedFS container): `exists()`/`getFileStatus()` on `s3://bucket` and `s3://bucket/` return a directory status; missing-bucket root → `exists()` false / `FileNotFoundException`; `open(root)` and `delete(root, recursive)` fail with a clear error and leave bucket contents intact. - Red-green verified: without the fix, the root-path tests fail with the exact `Key cannot be empty` marshalling stack from the JIRA. - Rest of the module's suites pass (filesystem, recoverable-writer recovery, HA job run, metrics ITCases; unit tests, checkstyle, spotless). ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: yes ## Documentation - Does this pull request introduce a new feature? no --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: opencode 1.18.34 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
