Samrat002 opened a new pull request, #27788:
URL: https://github.com/apache/flink/pull/27788
## What is the purpose of the change
This pull request adds **bucket-level configuration support** to the Native
S3 FileSystem, enabling different S3 buckets to use different connection
settings (path-style access, endpoint, credentials, SSE, assume-role). This
aligns with the Hadoop S3 FileSystem and supports use cases where checkpoints
and savepoints use different buckets with different requirements.
## Brief change log
- Added `BucketConfigProvider` to parse bucket-specific configuration from
Flink config using the format `s3.bucket.<bucket-name>.<property>`
- Added `S3BucketConfig` to hold bucket-level settings (endpoint,
path-style, credentials, SSE, assume-role)
- Added `S3ClientProviderCache` to cache S3 client providers per bucket
configuration
- Updated `NativeS3FileSystem` to use bucket-specific clients when
configured, otherwise the default client
- Updated `NativeS3FileSystemFactory` to create and pass
`BucketConfigProvider`, and added `CONNECTION_TIMEOUT` and `SOCKET_TIMEOUT`
options
- Documented new options in README: `s3.client.connection-timeout`,
`s3.client.socket-timeout`, and bucket-level configuration format
## Verifying this change
This change adds tests and can be verified as follows:
- Added `BucketConfigProviderTest` with tests for:
- Parsing bucket configs from the configuration
- Bucket config with encryption (SSE-KMS)
- Bucket config with assume-role
- Bucket names containing dots
- `S3BucketConfig` builder
Running a State job with a checkpointing bucket and a savepoint bucket
configured different
![Uploading Screenshot 2026-03-20 at 9.02.51 AM.png…]()
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: **no**
- The S3 file system connector: **yes**
## Documentation
- Does this pull request introduce a new feature? **yes**
- If yes, how is the feature documented? **README** – new configuration
options and bucket-level configuration format are documented in
`flink-s3-fs-native/README.md`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
