gaborgsomogyi opened a new pull request, #30043:
URL: https://github.com/apache/flink/pull/30043

   ## What is the purpose of the change
   
   A JobManager obtains its delegation tokens when the process starts. The 
periodic obtain of new tokens was started by the `ResourceManager` of a leader 
session and stopped with it. A standby JobManager, and a JobManager which lost 
the leadership, therefore never obtains new tokens for the process-local 
receivers it holds, for example the S3 or HDFS credentials the high 
availability storage is read with.
   
   When such a JobManager becomes the leader after its tokens expired, the 
Dispatcher recovery reads the HA storage right away, while the 
`ResourceManager`, which would obtain new tokens, is still starting. The read 
fails with `Could not retrieve JobResults of globally-terminated jobs from 
JobResultStore`, which is a fatal error, and the new leader exits.
   
   This PR makes obtaining the tokens a property of the process and the 
distribution of the tokens to the TaskManagers a property of the leader session.
   
   ## Brief change log
   
     - `DelegationTokenManager` (`@Internal`): `start()` and `stop()` belong to 
the process, `attachListener()` and `detachListener()` to a leader session. 
`start(Listener)` and the no-argument `obtainDelegationTokens()` are removed, 
the overload with a `DelegationTokenContainer` stays (YARN client).
     - `DefaultDelegationTokenManager`: one chain of obtains per process 
(`CREATED -> RUNNING -> STOPPED`). `start()` obtains inline and fails fast, as 
the one-time obtain did. `attachListener()` hands the newest tokens to the 
listener without contacting the token services. The callbacks to the listener 
are serialized, so it never gets older tokens after newer ones. 
`detachListener()` waits for the callback in progress, so none runs after it 
returns (bounded by the RPC timeout of the TaskManager update). The `Clock` can 
be injected.
     - `ResourceManager` attaches and detaches itself instead of starting and 
stopping the periodic obtain.
     - `ClusterEntrypoint` and `MiniCluster` start the manager where they 
obtained the tokens once, and stop it when they shut down. 
`ClusterEntrypoint#createDelegationTokenManager` is protected so that the test 
can see the calls. A failing `stop()` of the manager is collected like the 
failures of the other services, so the remaining services are stopped and the 
termination future of the entrypoint completes.
     - `security-delegation-token.md`: section "Delegation Tokens and High 
Availability".
     - Naming (second commit, no behavior change): the manager never renews a 
token, it obtains new ones before the old ones expire. The names, Javadoc and 
log texts of `DefaultDelegationTokenManager` say obtain and expiry 
(`validUntil`) instead of renewal and next renewal. The keys of the config 
options keep their "renewal" wording.
     - `DelegationTokenProviderRepository` (new, `@Internal`, the counterpart 
of `DelegationTokenReceiverRepository`): the provider handling moved out of 
`DefaultDelegationTokenManager` unchanged (loading, the enabled flag, init, 
duplicate and prefix checks, obtaining the tokens and their expiry). The 
manager keeps the lifecycle, the scheduling, the listener, the check that every 
provider has a receiver, and `obtainDelegationTokensAndGetExpiry` as the seam 
for tests. The log lines of the provider handling come from the new logger 
(third commit, no behavior change).
   
   Behavior changes:
   
     - Standby JobManagers keep obtaining new delegation tokens, so they 
contact the configured token services as well. N JobManagers issue N times as 
many tokens.
     - The start of a leader session does not obtain tokens any more, it uses 
the tokens its process holds. Providers which report no `validUntil` obtain 
tokens only when the process starts.
     - An `Error` of a provider during an obtain now also ends a standby 
JobManager, as it ended the leader before.
   
   ## Verifying this change
   
   Added new automated tests + triggered HA failover on real cluster.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? not applicable
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (Claude Code, see the `Generated-by` trailer of the commit)
   
   Generated-by: Claude Code with Claude Sonnet 5.5
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to