gaborgsomogyi opened a new pull request, #21694: URL: https://github.com/apache/flink/pull/21694
## What is the purpose of the change At the moment S3 authentication works in a way that each and every TaskManager logs in using long lived credentials. This puts quite some load to the S3 servers. In this PR I've added session token support. Namely: * one provides long lived credentials * JobManager logs in to S3 and obtains temporary session credentials * Flink propagates temporary session credentials to all the TMs * All TMs are using the short lived credentials This decreases the attack surface and removes some load from the S3 servers. ## Brief change log * Moved `DelegationTokenProvider` and `DelegationTokenReceiver` to core * Added `S3DelegationTokenProvider`, `S3DelegationTokenReceiver` and `DynamicTemporaryAWSCredentialsProvider` to handle session credentials * `DefaultDelegationTokenManager` now loads providers/receivers from plugins too * `DefaultPluginManager` now uses a single class loader within a plugin * New/modified tests ## Verifying this change Existing/new unit/integration tests + manually on [minikube](https://gist.github.com/gaborgsomogyi/ac4f71ead8494da2f5c35265bcb1e885). ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: yes ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
