gaborgsomogyi opened a new pull request, #30043:
URL: https://github.com/apache/flink/pull/30043
## What is the purpose of the change
A JobManager obtains its delegation tokens when the process starts. The
periodic obtain of new tokens was started by the `ResourceManager` of a leader
session and stopped with it. A standby JobManager, and a JobManager which lost
the leadership, therefore never obtains new tokens for the process-local
receivers it holds, for example the S3 or HDFS credentials the high
availability storage is read with.
When such a JobManager becomes the leader after its tokens expired, the
Dispatcher recovery reads the HA storage right away, while the
`ResourceManager`, which would obtain new tokens, is still starting. The read
fails with `Could not retrieve JobResults of globally-terminated jobs from
JobResultStore`, which is a fatal error, and the new leader exits.
This PR makes obtaining the tokens a property of the process and the
distribution of the tokens to the TaskManagers a property of the leader session.
## Brief change log
- `DelegationTokenManager` (`@Internal`): `start()` and `stop()` belong to
the process, `attachListener()` and `detachListener()` to a leader session.
`start(Listener)` and the no-argument `obtainDelegationTokens()` are removed,
the overload with a `DelegationTokenContainer` stays (YARN client).
- `DefaultDelegationTokenManager`: one chain of obtains per process
(`CREATED -> RUNNING -> STOPPED`). `start()` obtains inline and fails fast, as
the one-time obtain did. `attachListener()` hands the newest tokens to the
listener without contacting the token services. The callbacks to the listener
are serialized, so it never gets older tokens after newer ones.
`detachListener()` waits for the callback in progress, so none runs after it
returns (bounded by the RPC timeout of the TaskManager update). The `Clock` can
be injected.
- `ResourceManager` attaches and detaches itself instead of starting and
stopping the periodic obtain.
- `ClusterEntrypoint` and `MiniCluster` start the manager where they
obtained the tokens once, and stop it when they shut down.
`ClusterEntrypoint#createDelegationTokenManager` is protected so that the test
can see the calls. A failing `stop()` of the manager is collected like the
failures of the other services, so the remaining services are stopped and the
termination future of the entrypoint completes.
- `security-delegation-token.md`: section "Delegation Tokens and High
Availability".
- Naming (second commit, no behavior change): the manager never renews a
token, it obtains new ones before the old ones expire. The names, Javadoc and
log texts of `DefaultDelegationTokenManager` say obtain and expiry
(`validUntil`) instead of renewal and next renewal. The keys of the config
options keep their "renewal" wording.
- `DelegationTokenProviderRepository` (new, `@Internal`, the counterpart
of `DelegationTokenReceiverRepository`): the provider handling moved out of
`DefaultDelegationTokenManager` unchanged (loading, the enabled flag, init,
duplicate and prefix checks, obtaining the tokens and their expiry). The
manager keeps the lifecycle, the scheduling, the listener, the check that every
provider has a receiver, and `obtainDelegationTokensAndGetExpiry` as the seam
for tests. The log lines of the provider handling come from the new logger
(third commit, no behavior change).
Behavior changes:
- Standby JobManagers keep obtaining new delegation tokens, so they
contact the configured token services as well. N JobManagers issue N times as
many tokens.
- The start of a leader session does not obtain tokens any more, it uses
the tokens its process holds. Providers which report no `validUntil` obtain
tokens only when the process starts.
- An `Error` of a provider during an obtain now also ends a standby
JobManager, as it ended the leader before.
## Verifying this change
Added new automated tests + triggered HA failover on real cluster.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude Code, see the `Generated-by` trailer of the commit)
Generated-by: Claude Code with Claude Sonnet 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]