[ 
https://issues.apache.org/jira/browse/FLINK-41435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

LuChengxiang updated FLINK-41435:
---------------------------------
    Description: 
h2. Summary

Flink periodically obtains HDFS delegation tokens and distributes them through 
HadoopDelegationTokenReceiver. The receiver adds refreshed credentials to 
UserGroupInformation.getCurrentUser() of the thread running the refresh. A 
long-lived Hadoop client, especially a checkpoint-storage DFSClient, may 
instead use the UGI/Subject captured when that client was created. If those 
identities differ, the token refresh succeeds and the receiver reports success, 
but the new credentials are not delivered to the client's UGI. The client 
continues using its startup token and fails at the 24h HDFS token deadline with 
SecretManager$InvalidToken.

In the observed Kubernetes application-mode reproduction, the mismatch becomes 
reachable through ForkJoinPool compensation. The provider blocks in 
{{CompletableFuture.get()}} on a ForkJoinWorkerThread; FJP creates dispatcher 
workers while {{freshUGI.doAs(...)}} is effective, and those workers inherit 
the fresh Subject. A later {{cluster-io}} worker created by one of them 
inherits the same Subject and can run a token receiver under it.

The defect is receiver target selection, not token acquisition or client 
binding. The proposed fix targets the UGI held by HadoopSecurityContext, 
falling back to the login UGI, instead of a pooled worker's current UGI. Here 
"UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means the 
identity it carries; identity comparisons use Subject identity.

h2. Observed failure

This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The 
primary failure was in the checkpoint storage path:

{noformat}
CheckpointCoordinator.initializeCheckpointLocation
  -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
  -> HadoopFileSystem.mkdirs
  -> DFSClient.mkdirs
{noformat}

After the 24h deadline, the checkpoint-storage client reported InvalidToken for 
its startup token even though refresh had obtained a new HDFS token and the 
receiver reported success.

Not every Kubernetes application-mode job is affected. The failure is 
timing-dependent: a long-lived Hadoop client must capture one Subject/UGI 
lifecycle, and a later refresh receiver must run under a different one.

h2. Root cause

The delivery bug is in HadoopDelegationTokenReceiver. It targets 
{{UserGroupInformation.getCurrentUser()}} of the refresh thread. 
{{getCurrentUser()}} resolves from the thread's effective AccessControlContext 
and is not guaranteed to be the UGI held by HadoopSecurityContext or the UGI 
captured by a long-lived checkpoint-storage DFSClient. In the observed failure, 
the checkpoint client's captured UGI had the login/HadoopSecurityContext 
Subject, while the receiver could run under a pooled worker's inherited fresh 
Subject.

h3. Acquisition and refresh phases

Token delivery in the JobManager has three phases. Within each phase, the 
provider and the JM-local receiver run on the same token-manager thread.

* Bootstrap acquisition: {{ClusterEntrypoint.initializeServices()}} calls 
{{DefaultDelegationTokenManager.obtainDelegationTokens()}} on {{main}} before 
BlobServer starts. It updates local receivers for early external filesystem 
access. The receiver runs after {{freshUGI.doAs(...)}} returns, when current is 
login, so this phase is safe.
* ResourceManager startup acquisition: {{ResourceManager.onStart()}} -> 
{{DefaultDelegationTokenManager.start()}} -> {{startTokensUpdate()}} runs on 
{{default-dispatcher}}. This provider call is where the observed ForkJoinPool 
expansion occurs, but the startup receiver itself still runs on 
{{default-dispatcher}} after the provider returns, so it does not target the 
fresh Subject.
* Scheduled refresh: later updates are submitted through {{cluster-io}}. A 
receiver running on a {{cluster-io}} worker that inherited the fresh Subject 
resolves {{getCurrentUser()}} to that Subject and can target the wrong UGI.

TaskManager receivers are invoked by {{TaskExecutor}} token updates, not on 
{{cluster-io}}; the probe in this issue covers the JM-local receiver path only. 
These two startup token acquisitions are separate from the double 
{{SecurityUtils.install}} sequence discussed under Scope and boundaries.

h3. How the fresh Subject reaches a pooled worker

The observed window is the ResourceManager startup acquisition. The provider 
runs on {{default-dispatcher-12}}, a ForkJoinWorkerThread. The fresh Subject 
comes from {{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls 
{{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}. 
That creates a new UGI for the same principal and does not replace the process 
login user.

While {{freshUGI.doAs(...)}} is effective, {{fs.addDelegationTokens(...)}} 
performs a blocking HDFS RPC in {{CompletableFuture.get()}}. ForkJoinPool 
enters {{managedBlock()}}/{{tryCompensate()}} and creates 
{{default-dispatcher-13}}, {{default-dispatcher-14}}, and 
{{default-dispatcher-15}}.

The Thread constructor captures the effective AccessControlContext returned by 
{{AccessController.getContext()}} into the new thread's 
{{inheritedAccessControlContext}}. This is a stack-derived snapshot of the 
effective context, not the entire live Java stack, and it includes the Subject 
active during {{doAs}}. After {{doAs}} returns, current on 
{{default-dispatcher-12}} resolves to login again, but the newly created 
workers retain the fresh Subject.

{{default-dispatcher-15}} later creates {{cluster-io-thread-3}}. On that 
worker, current Subject is the fresh Subject, while login and 
HadoopSecurityContext Subjects remain the login Subject. A receiver invocation 
on this worker resolves {{getCurrentUser()}} to the fresh Subject and writes 
credentials there, not to the UGI captured by the long-lived client.

In the probe block, HadoopSecurityContext Subject means the Subject of the UGI 
held by the installed HadoopSecurityContext; it matches the login Subject here, 
but that is not guaranteed in general.

{noformat}
provider thread                     = default-dispatcher-12
freshUGI Subject                    = 817147333
login/HadoopSecurityContext Subject = 1333938290

current Subject before doAs         = 1333938290
current Subject inside doAs         = 817147333
current Subject after doAs          = 1333938290

FJP workers created during the provider's blocked RPC:
  default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12

later cluster-io worker:
  cluster-io-thread-3, creator=default-dispatcher-15
  current Subject                    = 817147333
  login/HadoopSecurityContext Subject = 1333938290
  JM-local receiver target (getCurrentUser()) = 817147333
{noformat}

This is a per-thread inherited security context, not a global mutation of 
UserGroupInformation. It is a receiver target-selection mismatch, not a 
client-binding error. Clients created under an unrelated transient Subject 
require separate rebinding or recreation and are out of scope.

h2. Proposed change

This issue fixes token delivery for the observed long-lived-client case, 
primarily checkpoint storage.

* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, select a single target UGI: use the UGI 
held by HadoopSecurityContext when present; otherwise fall back to 
{{UserGroupInformation.getLoginUser()}}. Add refreshed credentials only to that 
UGI. Do not target {{UserGroupInformation.getCurrentUser()}}, which may be a 
pooled worker's inherited Subject.
* Add tests covering (a) the HadoopSecurityContext UGI takes precedence over 
login/current, (b) the login UGI is used when no HadoopSecurityContext is 
installed, and (c) the inherited current UGI is not updated.

The change makes the delivery target deterministic. It does not rebind a client 
already created under a different Subject.

h2. Scope and boundaries

h3. Out of scope

* Preventing HDFS client initialization or blocking RPCs from occurring inside 
{{freshUGI.doAs(...)}} or the provider's ForkJoinPool worker.
* Closing or rebinding FileSystem instances created under a transient fresh 
Subject; this is a separate client-lifecycle cleanup.
* Delivering credentials to clients created under an unrelated transient 
Subject; those clients require separate rebinding or recreation.
* Changing the provider's {{getCurrentUser()}} usage. 
{{HadoopFSDelegationTokenProvider}} already acquires tokens inside 
{{KerberosLoginProvider.doLoginAndReturnUGI()}}; its remaining 
{{getCurrentUser()}} use is a token-required pre-check and renewer-name lookup. 
No provider change is required for the observed receiver mismatch; it remains a 
separate follow-up if provider-side behavior is observed.
* Double security installation and {{SecurityUtils.install}} idempotency. A 
diagnostic build where the second installation was skipped still failed at the 
24h deadline. The early/duplicate installation adds lifecycle complexity, but 
the probe does not establish it as a cause of the receiver mismatch.

h3. YARN

YARN is not a confirmed reproduction in this issue. In one controlled YARN 
application-mode probe, the provider also ran on a Pekko {{default-dispatcher}} 
worker, but its provider window stayed at {{poolSize=3}} / {{active=2}} with no 
{{tryCompensate}} or {{createWorker}}, so no new dispatcher worker was created 
during the provider's fresh-Subject window.

YARN did show ForkJoinPool expansion during the separate JobGraph-generation 
startup path, so this is not evidence that YARN is immune. The sample is 
limited: the provider's {{CompletableFuture}} state was not instrumented, so 
the YARN trigger is not established.

h3. Versions

Before 1.17, the legacy {{HadoopModule}} path added credentials directly to the 
login user; the {{HadoopDelegationTokenReceiver}} / 
{{UserGroupInformation.getCurrentUser()}} path starts with 1.17. 1.18, 1.19, 
and 1.20 share that receiver behavior. 1.18 is not claimed to be immune.

h2. Testing

h3. Observed probe evidence

* A controlled Kubernetes application-mode probe recorded the chain from 
{{fs.addDelegationTokens()}} through Hadoop ipc {{Client.getRpcResponse()}}, 
{{CompletableFuture.get()}}, {{ForkJoinPool.managedBlock()}}, 
{{tryCompensate()}}, and Pekko {{ThreadFactory.newThread()}}. The creator and 
the new dispatcher workers had the same fresh Subject identity.
* The same probe recorded {{cluster-io-thread-3}} created by 
{{default-dispatcher-15}} with {{subjectId=817147333}}, while the 
login/HadoopSecurityContext Subject remained {{1333938290}}.
* A later JM-local receiver invocation on {{cluster-io-thread-3}} observed 
current Subject=817147333 while login/HadoopSecurityContext Subject=1333938290. 
The pre-fix receiver therefore targets the fresh UGI on this worker; 
TaskManager receiver behavior is not covered by this probe and requires a 
separate TaskManager receiver test.

h3. Verification

* Add receiver unit tests for (a) HadoopSecurityContext UGI precedence with 
login/current not updated, (b) login fallback when no HadoopSecurityContext is 
installed, and (c) the inherited current UGI not targeted.
* Trigger multiple post-startup token refreshes in a long-running job and 
verify which executor runs the provider, whether new ForkJoinPool workers are 
created, and which Subject the receiver's current user resolves to.
* Repeat the YARN provider-window probe while instrumenting the provider's 
{{CompletableFuture}} state; the current YARN sample is summarized under Scope 
and boundaries.
* A deployed application-mode job should survive the 24h token deadline and 
continue checkpointing without SecretManager$InvalidToken. Verify that the 
checkpoint-storage DFSClient receives the new token.

h2. Acceptance criteria

* A token refresh updates the UGI held by HadoopSecurityContext when present, 
otherwise the login UGI, and does not use a pooled worker's inherited current 
UGI; a checkpoint-storage client bound to the selected UGI receives the new 
token.
* A long-running application-mode job survives the 24h HDFS token deadline.


  was:
h2. Summary

Flink periodically obtains HDFS delegation tokens and distributes them through 
HadoopDelegationTokenReceiver. The receiver currently adds new credentials to 
UserGroupInformation.getCurrentUser() on the thread that performs the refresh. 
Long-lived Hadoop clients, especially checkpoint storage, can capture a 
UGI/Subject when their DFSClient is created and use that captured identity for 
later RPC connections.

If the refresh path resolves to a different current UGI from the one captured 
by a long-lived client, the client can continue using its startup token even 
though a new token was obtained successfully. The failure appears at the 24h 
HDFS token deadline as SecretManager$InvalidToken. The key window is 
ForkJoinPool compensation: when the provider blocks in 
{{CompletableFuture.get()}} on a ForkJoinWorkerThread, FJP may create new 
dispatcher workers while {{freshUGI.doAs(...)}} is effective; those workers 
inherit the fresh Subject.

The observed Kubernetes reproduction is in the ResourceManager startup 
acquisition path, not the bootstrap acquisition or a later scheduled refresh. 
The receiver mismatch is not limited to that path: it applies whenever a 
refresh task resolves to a different current UGI. In this issue, "UGI" means 
the Hadoop UserGroupInformation wrapper, while "Subject" means the identity 
carried by that UGI; identity comparisons between wrappers use Subject identity.

h2. Observed failure

This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The 
primary failure was in the checkpoint storage path:

{noformat}
CheckpointCoordinator.initializeCheckpointLocation
  -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
  -> HadoopFileSystem.mkdirs
  -> DFSClient.mkdirs
{noformat}

The checkpoint storage client reported InvalidToken for the startup token after 
the refresh had created a new HDFS token and the receiver reported success.

Not every Kubernetes application-mode job is affected. The failure is 
timing-dependent: it requires a long-lived Hadoop client to have captured one 
Subject/UGI lifecycle while the refresh path later resolves to a different one.

Installation idempotency is out of scope. A diagnostic build that skipped the 
second security installation still failed at the 24h HDFS token deadline. The 
Kubernetes application entrypoint (1.19+) installs security early for artifact 
fetching and then installs again in {{startCluster}}; this creates an 
additional security-module/context lifecycle, but the probe does not establish 
it as a cause of the receiver mismatch. Flink's {{FileSystem.initialize}} 
clears only Flink's {{FileSystem}} cache; it does not clear Hadoop 
{{FileSystem}}/{{DFSClient}} caches or rebind existing UGI/Subject-held clients.

h2. Root cause

HadoopDelegationTokenReceiver targets {{UserGroupInformation.getCurrentUser()}} 
of the thread executing the refresh. That current UGI is resolved from the 
thread's effective AccessControlContext and is not guaranteed to be the UGI 
held by {{HadoopSecurityContext}} or the UGI captured by a long-lived 
checkpoint-storage DFSClient. For the observed checkpoint client, its captured 
UGI has the login/HadoopSecurityContext Subject; the receiver's target 
selection is therefore the delivery bug.

h3. Acquisition phases

The startup path has two acquisitions, followed by scheduled refreshes:

* Bootstrap acquisition: {{ClusterEntrypoint.initializeServices()}} calls 
{{DefaultDelegationTokenManager.obtainDelegationTokens()}} on {{main}} before 
BlobServer starts. It updates only local receivers for early external 
filesystem access.
* ResourceManager startup acquisition: {{ResourceManager.onStart()}} -> 
{{DefaultDelegationTokenManager.start()}} -> {{startTokensUpdate()}} runs on 
the default-dispatcher.
* Scheduled refresh: later updates are submitted through the {{cluster-io}} 
executor.

For each phase, the provider and the JM-local receiver run on the same 
token-manager thread. The bootstrap receiver is safe: it runs on {{main}} after 
{{freshUGI.doAs(...)}} returns, when current is login. The ResourceManager 
startup receiver also remains on default-dispatcher; only the later scheduled 
refresh runs on {{cluster-io}}. TaskManager receivers are invoked by 
{{TaskExecutor}} token updates, not on {{cluster-io}}.

h3. FJP window and propagation

The ResourceManager startup acquisition is the window. The provider runs on a 
ForkJoinWorkerThread; {{fs.addDelegationTokens(...)}} performs a blocking HDFS 
RPC in {{CompletableFuture.get()}}; FJP enters 
{{managedBlock()}}/{{tryCompensate()}} and creates new dispatcher workers while 
the fresh Subject is active. The probe observed:

{noformat}
provider thread                 = default-dispatcher-12
freshUGI Subject                = 817147333
login/HadoopSecurityContext Subject = 1333938290

current Subject before doAs    = 1333938290
current Subject inside doAs    = 817147333
current Subject after doAs     = 1333938290

FJP workers created during the provider's blocked RPC:
  default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12

later cluster-io worker:
  cluster-io-thread-3, creator=default-dispatcher-15
  current Subject                 = 817147333
  login/HadoopSecurityContext Subject = 1333938290
  JM-local receiver target (getCurrentUser()) = 817147333
{noformat}

Here {{HadoopSecurityContext Subject}} means the Subject of the UGI held by the 
installed {{HadoopSecurityContext}}. In this probe it is the same Subject 
identity as the login user; that is not guaranteed in general.

The fresh Subject is not created by HadoopSecurityContext or by the dispatcher. 
It belongs to the UGI returned by 
{{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls 
{{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}. 
That call performs a Kerberos login and returns a newly constructed UGI for the 
same principal; it does not replace the process login user.

During {{freshUGI.doAs(...)}}, the Thread constructor captures the 
{{AccessControlContext}} snapshot returned by 
{{AccessController.getContext()}}. That snapshot reflects the current thread's 
effective, stack-derived context, including the active {{Subject.doAs(...)}} 
Subject; it is not the entire live Java stack and not merely a reference to one 
stack frame. The new thread stores this snapshot in its 
{{inheritedAccessControlContext}}. After {{doAs}} returned, current on the 
creator resolved to the login Subject again, but the new workers retained the 
inherited fresh Subject.

A dispatcher worker created during the {{doAs}} window later created 
{{cluster-io-thread-3}}. On {{cluster-io-thread-3}}, current Subject was the 
fresh Subject, while login and HadoopSecurityContext Subjects remained 
{{1333938290}}. If a receiver invocation runs on this worker, the pre-fix 
receiver calls {{UserGroupInformation.getCurrentUser().addCredentials(...)}}. 
That resolves to a UGI whose Subject is the fresh Subject, so the credentials 
are written to that UGI's Subject rather than to the UGI captured by the 
long-lived client. This is a receiver target-selection mismatch, not a 
client-binding error. For the observed checkpoint client, its captured UGI has 
the login/HadoopSecurityContext Subject; the proposed receiver targets that 
UGI. Clients bound to an unrelated transient Subject remain out of scope. This 
is a per-thread inherited security context, not a global mutation of 
{{UserGroupInformation}}. The probe covers only the JobManager receiver path; 
TaskManager receivers receive token updates through the ResourceManager.


h2. YARN status and scope

In a controlled YARN application-mode probe, the provider also ran on a Pekko 
default-dispatcher worker. Its provider window stayed at {{poolSize=3}} / 
{{active=2}} with no {{tryCompensate}} or {{createWorker}}, so no new 
dispatcher worker was created while the provider's fresh Subject was active. 
YARN did show ForkJoinPool expansion during the separate JobGraph-generation 
startup path, so this is not evidence that YARN is immune.

This is one YARN sample; the provider's {{CompletableFuture}} state was not 
instrumented, so the YARN trigger is not established. YARN is not listed as a 
confirmed reproduction, but it is not claimed to be exempt.

h2. Version context

Before 1.17, the legacy {{HadoopModule}} path added credentials directly to the 
login user; the {{HadoopDelegationTokenReceiver}} / 
{{UserGroupInformation.getCurrentUser()}} path starts with 1.17. 1.18, 1.19, 
and 1.20 share that receiver behavior. 1.18 is not claimed to be immune.

h2. Proposed change

This issue addresses the case where a token is obtained successfully but the 
refreshed credentials do not reach long-lived Hadoop clients, primarily 
checkpoint storage.

* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, select a single target UGI: use the UGI 
held by HadoopSecurityContext when present; otherwise fall back to 
{{UserGroupInformation.getLoginUser()}}. Add refreshed credentials only to that 
UGI. Do not target {{UserGroupInformation.getCurrentUser()}}, which may be a 
pooled worker's inherited Subject.
* Add tests covering (a) the HadoopSecurityContext UGI takes precedence over 
login/current, (b) the login UGI is used when no HadoopSecurityContext is 
installed, and (c) the inherited current UGI is not updated.

Out of scope for this issue:

* Preventing HDFS client initialization or blocking RPCs from occurring inside 
{{freshUGI.doAs(...)}} or the provider's ForkJoinPool worker.
* Closing or rebinding FileSystem instances created under a transient fresh 
Subject; this is a separate client-lifecycle cleanup.
* Delivering credentials to clients created under an unrelated transient 
Subject; those clients require separate rebinding or recreation.
* Changing the provider's {{getCurrentUser()}} usage. 
{{HadoopFSDelegationTokenProvider}} already acquires tokens inside 
{{KerberosLoginProvider.doLoginAndReturnUGI()}}; its remaining 
{{getCurrentUser()}} use is a token-required pre-check and renewer-name lookup. 
No provider change is required for the observed receiver mismatch; it remains a 
separate follow-up if provider-side behavior is observed.
* SecurityUtils.install idempotency.

The proposed receiver change does not rely solely on the refresh thread's 
inherited Subject. It does not by itself guarantee that a client already 
created under a different Subject is rebound. That remains a separate lifecycle 
concern.

h2. Testing

* A controlled Kubernetes application-mode probe recorded the chain from 
{{fs.addDelegationTokens()}} through Hadoop ipc {{Client.getRpcResponse()}}, 
{{CompletableFuture.get()}}, {{ForkJoinPool.managedBlock()}}, 
{{tryCompensate()}}, and Pekko {{ThreadFactory.newThread()}}. The creator and 
the new dispatcher workers had the same fresh Subject identity.
* The same probe recorded {{cluster-io-thread-3}} created by 
{{default-dispatcher-15}} with {{subjectId=817147333}}, while the 
login/HadoopSecurityContext Subject remained {{1333938290}}.
* A later JM-local receiver invocation on {{cluster-io-thread-3}} observed 
current Subject=817147333 while login/HadoopSecurityContext Subject=1333938290. 
The pre-fix receiver therefore targets the fresh UGI on this worker; 
TaskManager receiver behavior is not covered by this probe and requires a 
separate TaskManager receiver test.
* Add receiver unit tests for (a) HadoopSecurityContext UGI precedence with 
login/current not updated, (b) login fallback when no HadoopSecurityContext is 
installed, and (c) the inherited current UGI not targeted.
* Trigger multiple post-startup token refreshes in a long-running job and 
verify which executor runs the provider, whether new ForkJoinPool workers are 
created, and which Subject the receiver's current user resolves to.
* Repeat the YARN provider-window probe while instrumenting the provider's 
{{CompletableFuture}} state; the current YARN sample is summarized in "YARN 
status and scope".
* A deployed application-mode job should survive the 24h token deadline and 
continue checkpointing without SecretManager$InvalidToken. Verify that the 
checkpoint-storage DFSClient receives the new token.

h2. Acceptance criteria

* A token refresh updates the UGI held by HadoopSecurityContext when present, 
otherwise the login UGI, and does not use a pooled worker's inherited current 
UGI; a checkpoint-storage client bound to the selected UGI receives the new 
token.
* A long-running application-mode job survives the 24h HDFS token deadline.



> Hadoop delegation token refresh can target a different UGI than Hadoop clients
> ------------------------------------------------------------------------------
>
>                 Key: FLINK-41435
>                 URL: https://issues.apache.org/jira/browse/FLINK-41435
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Configuration
>    Affects Versions: 2.0.0, 1.19.0, 1.20.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0
>         Environment: Flink 1.19.0 and later (1.19.x, 1.20.x, 2.0 through 
> 2.4), plus master (2.5-SNAPSHOT); Kubernetes and standalone application mode 
> with Kerberos and HDFS delegation tokens. The mismatch is more likely when 
> FileSystemJobResultStore and the token manager share the cluster-io executor 
> (FLINK-27204).
>            Reporter: LuChengxiang
>            Priority: Major
>
> h2. Summary
> Flink periodically obtains HDFS delegation tokens and distributes them 
> through HadoopDelegationTokenReceiver. The receiver adds refreshed 
> credentials to UserGroupInformation.getCurrentUser() of the thread running 
> the refresh. A long-lived Hadoop client, especially a checkpoint-storage 
> DFSClient, may instead use the UGI/Subject captured when that client was 
> created. If those identities differ, the token refresh succeeds and the 
> receiver reports success, but the new credentials are not delivered to the 
> client's UGI. The client continues using its startup token and fails at the 
> 24h HDFS token deadline with SecretManager$InvalidToken.
> In the observed Kubernetes application-mode reproduction, the mismatch 
> becomes reachable through ForkJoinPool compensation. The provider blocks in 
> {{CompletableFuture.get()}} on a ForkJoinWorkerThread; FJP creates dispatcher 
> workers while {{freshUGI.doAs(...)}} is effective, and those workers inherit 
> the fresh Subject. A later {{cluster-io}} worker created by one of them 
> inherits the same Subject and can run a token receiver under it.
> The defect is receiver target selection, not token acquisition or client 
> binding. The proposed fix targets the UGI held by HadoopSecurityContext, 
> falling back to the login UGI, instead of a pooled worker's current UGI. Here 
> "UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means the 
> identity it carries; identity comparisons use Subject identity.
> h2. Observed failure
> This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
> ZooKeeper-based high availability and HDFS-backed checkpoint storage. The 
> primary failure was in the checkpoint storage path:
> {noformat}
> CheckpointCoordinator.initializeCheckpointLocation
>   -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
>   -> HadoopFileSystem.mkdirs
>   -> DFSClient.mkdirs
> {noformat}
> After the 24h deadline, the checkpoint-storage client reported InvalidToken 
> for its startup token even though refresh had obtained a new HDFS token and 
> the receiver reported success.
> Not every Kubernetes application-mode job is affected. The failure is 
> timing-dependent: a long-lived Hadoop client must capture one Subject/UGI 
> lifecycle, and a later refresh receiver must run under a different one.
> h2. Root cause
> The delivery bug is in HadoopDelegationTokenReceiver. It targets 
> {{UserGroupInformation.getCurrentUser()}} of the refresh thread. 
> {{getCurrentUser()}} resolves from the thread's effective 
> AccessControlContext and is not guaranteed to be the UGI held by 
> HadoopSecurityContext or the UGI captured by a long-lived checkpoint-storage 
> DFSClient. In the observed failure, the checkpoint client's captured UGI had 
> the login/HadoopSecurityContext Subject, while the receiver could run under a 
> pooled worker's inherited fresh Subject.
> h3. Acquisition and refresh phases
> Token delivery in the JobManager has three phases. Within each phase, the 
> provider and the JM-local receiver run on the same token-manager thread.
> * Bootstrap acquisition: {{ClusterEntrypoint.initializeServices()}} calls 
> {{DefaultDelegationTokenManager.obtainDelegationTokens()}} on {{main}} before 
> BlobServer starts. It updates local receivers for early external filesystem 
> access. The receiver runs after {{freshUGI.doAs(...)}} returns, when current 
> is login, so this phase is safe.
> * ResourceManager startup acquisition: {{ResourceManager.onStart()}} -> 
> {{DefaultDelegationTokenManager.start()}} -> {{startTokensUpdate()}} runs on 
> {{default-dispatcher}}. This provider call is where the observed ForkJoinPool 
> expansion occurs, but the startup receiver itself still runs on 
> {{default-dispatcher}} after the provider returns, so it does not target the 
> fresh Subject.
> * Scheduled refresh: later updates are submitted through {{cluster-io}}. A 
> receiver running on a {{cluster-io}} worker that inherited the fresh Subject 
> resolves {{getCurrentUser()}} to that Subject and can target the wrong UGI.
> TaskManager receivers are invoked by {{TaskExecutor}} token updates, not on 
> {{cluster-io}}; the probe in this issue covers the JM-local receiver path 
> only. These two startup token acquisitions are separate from the double 
> {{SecurityUtils.install}} sequence discussed under Scope and boundaries.
> h3. How the fresh Subject reaches a pooled worker
> The observed window is the ResourceManager startup acquisition. The provider 
> runs on {{default-dispatcher-12}}, a ForkJoinWorkerThread. The fresh Subject 
> comes from {{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls 
> {{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}. 
> That creates a new UGI for the same principal and does not replace the 
> process login user.
> While {{freshUGI.doAs(...)}} is effective, {{fs.addDelegationTokens(...)}} 
> performs a blocking HDFS RPC in {{CompletableFuture.get()}}. ForkJoinPool 
> enters {{managedBlock()}}/{{tryCompensate()}} and creates 
> {{default-dispatcher-13}}, {{default-dispatcher-14}}, and 
> {{default-dispatcher-15}}.
> The Thread constructor captures the effective AccessControlContext returned 
> by {{AccessController.getContext()}} into the new thread's 
> {{inheritedAccessControlContext}}. This is a stack-derived snapshot of the 
> effective context, not the entire live Java stack, and it includes the 
> Subject active during {{doAs}}. After {{doAs}} returns, current on 
> {{default-dispatcher-12}} resolves to login again, but the newly created 
> workers retain the fresh Subject.
> {{default-dispatcher-15}} later creates {{cluster-io-thread-3}}. On that 
> worker, current Subject is the fresh Subject, while login and 
> HadoopSecurityContext Subjects remain the login Subject. A receiver 
> invocation on this worker resolves {{getCurrentUser()}} to the fresh Subject 
> and writes credentials there, not to the UGI captured by the long-lived 
> client.
> In the probe block, HadoopSecurityContext Subject means the Subject of the 
> UGI held by the installed HadoopSecurityContext; it matches the login Subject 
> here, but that is not guaranteed in general.
> {noformat}
> provider thread                     = default-dispatcher-12
> freshUGI Subject                    = 817147333
> login/HadoopSecurityContext Subject = 1333938290
> current Subject before doAs         = 1333938290
> current Subject inside doAs         = 817147333
> current Subject after doAs          = 1333938290
> FJP workers created during the provider's blocked RPC:
>   default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12
> later cluster-io worker:
>   cluster-io-thread-3, creator=default-dispatcher-15
>   current Subject                    = 817147333
>   login/HadoopSecurityContext Subject = 1333938290
>   JM-local receiver target (getCurrentUser()) = 817147333
> {noformat}
> This is a per-thread inherited security context, not a global mutation of 
> UserGroupInformation. It is a receiver target-selection mismatch, not a 
> client-binding error. Clients created under an unrelated transient Subject 
> require separate rebinding or recreation and are out of scope.
> h2. Proposed change
> This issue fixes token delivery for the observed long-lived-client case, 
> primarily checkpoint storage.
> * Add an accessor for the UGI held by HadoopSecurityContext.
> * In HadoopDelegationTokenReceiver, select a single target UGI: use the UGI 
> held by HadoopSecurityContext when present; otherwise fall back to 
> {{UserGroupInformation.getLoginUser()}}. Add refreshed credentials only to 
> that UGI. Do not target {{UserGroupInformation.getCurrentUser()}}, which may 
> be a pooled worker's inherited Subject.
> * Add tests covering (a) the HadoopSecurityContext UGI takes precedence over 
> login/current, (b) the login UGI is used when no HadoopSecurityContext is 
> installed, and (c) the inherited current UGI is not updated.
> The change makes the delivery target deterministic. It does not rebind a 
> client already created under a different Subject.
> h2. Scope and boundaries
> h3. Out of scope
> * Preventing HDFS client initialization or blocking RPCs from occurring 
> inside {{freshUGI.doAs(...)}} or the provider's ForkJoinPool worker.
> * Closing or rebinding FileSystem instances created under a transient fresh 
> Subject; this is a separate client-lifecycle cleanup.
> * Delivering credentials to clients created under an unrelated transient 
> Subject; those clients require separate rebinding or recreation.
> * Changing the provider's {{getCurrentUser()}} usage. 
> {{HadoopFSDelegationTokenProvider}} already acquires tokens inside 
> {{KerberosLoginProvider.doLoginAndReturnUGI()}}; its remaining 
> {{getCurrentUser()}} use is a token-required pre-check and renewer-name 
> lookup. No provider change is required for the observed receiver mismatch; it 
> remains a separate follow-up if provider-side behavior is observed.
> * Double security installation and {{SecurityUtils.install}} idempotency. A 
> diagnostic build where the second installation was skipped still failed at 
> the 24h deadline. The early/duplicate installation adds lifecycle complexity, 
> but the probe does not establish it as a cause of the receiver mismatch.
> h3. YARN
> YARN is not a confirmed reproduction in this issue. In one controlled YARN 
> application-mode probe, the provider also ran on a Pekko 
> {{default-dispatcher}} worker, but its provider window stayed at 
> {{poolSize=3}} / {{active=2}} with no {{tryCompensate}} or {{createWorker}}, 
> so no new dispatcher worker was created during the provider's fresh-Subject 
> window.
> YARN did show ForkJoinPool expansion during the separate JobGraph-generation 
> startup path, so this is not evidence that YARN is immune. The sample is 
> limited: the provider's {{CompletableFuture}} state was not instrumented, so 
> the YARN trigger is not established.
> h3. Versions
> Before 1.17, the legacy {{HadoopModule}} path added credentials directly to 
> the login user; the {{HadoopDelegationTokenReceiver}} / 
> {{UserGroupInformation.getCurrentUser()}} path starts with 1.17. 1.18, 1.19, 
> and 1.20 share that receiver behavior. 1.18 is not claimed to be immune.
> h2. Testing
> h3. Observed probe evidence
> * A controlled Kubernetes application-mode probe recorded the chain from 
> {{fs.addDelegationTokens()}} through Hadoop ipc {{Client.getRpcResponse()}}, 
> {{CompletableFuture.get()}}, {{ForkJoinPool.managedBlock()}}, 
> {{tryCompensate()}}, and Pekko {{ThreadFactory.newThread()}}. The creator and 
> the new dispatcher workers had the same fresh Subject identity.
> * The same probe recorded {{cluster-io-thread-3}} created by 
> {{default-dispatcher-15}} with {{subjectId=817147333}}, while the 
> login/HadoopSecurityContext Subject remained {{1333938290}}.
> * A later JM-local receiver invocation on {{cluster-io-thread-3}} observed 
> current Subject=817147333 while login/HadoopSecurityContext 
> Subject=1333938290. The pre-fix receiver therefore targets the fresh UGI on 
> this worker; TaskManager receiver behavior is not covered by this probe and 
> requires a separate TaskManager receiver test.
> h3. Verification
> * Add receiver unit tests for (a) HadoopSecurityContext UGI precedence with 
> login/current not updated, (b) login fallback when no HadoopSecurityContext 
> is installed, and (c) the inherited current UGI not targeted.
> * Trigger multiple post-startup token refreshes in a long-running job and 
> verify which executor runs the provider, whether new ForkJoinPool workers are 
> created, and which Subject the receiver's current user resolves to.
> * Repeat the YARN provider-window probe while instrumenting the provider's 
> {{CompletableFuture}} state; the current YARN sample is summarized under 
> Scope and boundaries.
> * A deployed application-mode job should survive the 24h token deadline and 
> continue checkpointing without SecretManager$InvalidToken. Verify that the 
> checkpoint-storage DFSClient receives the new token.
> h2. Acceptance criteria
> * A token refresh updates the UGI held by HadoopSecurityContext when present, 
> otherwise the login UGI, and does not use a pooled worker's inherited current 
> UGI; a checkpoint-storage client bound to the selected UGI receives the new 
> token.
> * A long-running application-mode job survives the 24h HDFS token deadline.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to