MartijnVisser opened a new pull request, #29823:
URL: https://github.com/apache/flink/pull/29823
## What is the purpose of the change
flink-s3-fs-hadoop bundles Hadoop 3.3.4, which fails on Java 23 and later
with `getSubject is not supported` (HADOOP-19212). hadoop-aws 3.4 requires AWS
SDK v2, so this moves the S3 filesystems to Hadoop 3.4.3 and flink-s3-fs-hadoop
to SDK v2. It continues #27026 by @ctrlaltdilj and builds on #29821, whose
commit is included until that one is merged.
## Brief change log
- The S3 delegation token carries SDK-independent credentials, and each
plugin creates the STS client with its own SDK.
- flink-s3-fs-hadoop uses SDK v2 for the recoverable writer. The FileSink
state format stays version 1, so flink-s3-fs-native can still restore it
(FLINK-40943).
- The recoverable writer rejects S3 part checksums
(`fs.s3a.create.checksum.algorithm`, `fs.s3a.checksum.generation`), because the
state doesn't store them.
- Common SDK v1 credentials provider names are mapped to SDK v2.
- flink-s3-fs-presto stays on SDK v1 and no longer bundles hadoop-aws, the
Analytics Accelerator and Netty.
- The classic S3A input stream stays the default.
- Docs and NOTICE files updated, including the third-party code that Netty
embeds.
## Verifying this change
- Unit tests for each plugin's STS client, the SDK v1 provider mapping
(resolved through Hadoop's `CredentialProviderListFactory`) and the version 1
serializer bytes. An ArchUnit test keeps the token classes SDK-free.
- The module tests and the recoverable writer ITCases against SeaweedFS
pass on Java 17, 21 (`-Pjava21-target`) and 25. On Java 25 the HA ITCases now
pass, while master fails them. `HadoopS3FileSystemBehaviorITCase` fails one
recursive-delete test on SeaweedFS; master fails two.
- The license checker reports no severe issues.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): yes
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no, the version 1 bytes are unchanged and pinned by a
test
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes, the FileSink
recovery path of flink-s3-fs-hadoop moves to AWS SDK v2
- The S3 file system connector: yes
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
