[ 
https://issues.apache.org/jira/browse/FLINK-41435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

LuChengxiang updated FLINK-41435:
---------------------------------
    Description: 
h2. Summary

Flink periodically obtains HDFS delegation tokens and distributes them through 
HadoopDelegationTokenReceiver. The receiver adds refreshed credentials to 
UserGroupInformation.getCurrentUser() of the thread running the refresh. A 
long-lived Hadoop client, especially a checkpoint-storage DFSClient, may 
instead use the UGI/Subject captured when that client was created. If those 
identities differ, the token refresh succeeds and the receiver reports success, 
but the new credentials are not delivered to the client's UGI. The client 
continues using its startup token and fails at the 24h HDFS token deadline with 
SecretManager$InvalidToken.

In the observed Kubernetes application-mode reproduction, the mismatch becomes 
reachable through ForkJoinPool's managed-block compensation. The provider 
blocks in {{CompletableFuture.get()}} on a ForkJoinWorkerThread; when no idle 
FJP thread is available and the pool has capacity, the compensation path 
creates additional dispatcher threads while {{freshUGI.doAs(...)}} is 
effective, and those threads inherit the fresh Subject. A later {{cluster-io}} 
thread created by one of them inherits the same Subject and can run a token 
receiver under it.

The defect is receiver target selection, not token acquisition or client 
binding. The proposed fix targets the UGI held by HadoopSecurityContext, 
falling back to the login UGI, instead of a pooled thread's current UGI. Here 
"UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means the 
identity it carries; identity comparisons use Subject identity.

h2. Observed failure

This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The 
primary failure was in the checkpoint storage path:

{noformat}
CheckpointCoordinator.initializeCheckpointLocation
  -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
  -> HadoopFileSystem.mkdirs
  -> DFSClient.mkdirs
{noformat}

After the 24h deadline, the checkpoint-storage client reported InvalidToken for 
its startup token even though refresh had obtained a new HDFS token and the 
receiver reported success.

Not every Kubernetes application-mode job is affected. The failure is 
timing-dependent: a long-lived Hadoop client must capture one Subject/UGI 
lifecycle, and a later refresh receiver must run under a different one.

h2. Root cause

The delivery bug is in HadoopDelegationTokenReceiver. It targets 
{{UserGroupInformation.getCurrentUser()}} of the refresh thread. 
{{getCurrentUser()}} resolves from the thread's effective AccessControlContext 
and is not guaranteed to be the UGI held by HadoopSecurityContext or the UGI 
captured by a long-lived checkpoint-storage DFSClient. In the observed failure, 
the checkpoint client's captured UGI had the login/HadoopSecurityContext 
Subject, while the receiver could run under a pooled thread's inherited fresh 
Subject.

h3. Acquisition and refresh phases

Token delivery in the JobManager has three phases. Within each phase, the 
provider and the JM-local receiver run on the same token-manager thread.

* Bootstrap acquisition: {{ClusterEntrypoint.initializeServices()}} calls 
{{DefaultDelegationTokenManager.obtainDelegationTokens()}} on {{main}} before 
BlobServer starts. It updates local receivers for early external filesystem 
access. The receiver runs after {{freshUGI.doAs(...)}} returns, when current is 
login, so this phase is safe.
* ResourceManager startup acquisition: {{ResourceManager.onStart()}} -> 
{{DefaultDelegationTokenManager.start()}} -> {{startTokensUpdate()}} runs on 
{{default-dispatcher}}. This provider call is where the observed ForkJoinPool 
expansion occurs, but the startup receiver itself still runs on 
{{default-dispatcher}} after the provider returns, so it does not target the 
fresh Subject.
* Scheduled refresh: later updates are submitted through {{cluster-io}}. A 
receiver running on a {{cluster-io}} thread that inherited the fresh Subject 
resolves {{getCurrentUser()}} to that Subject and can target the wrong UGI.

TaskManager receivers are invoked by {{TaskExecutor}} token updates, not on 
{{cluster-io}}; the probe in this issue covers the JM-local receiver path only. 
These two startup token acquisitions are separate from the double 
{{SecurityUtils.install}} sequence discussed under Scope and boundaries.

h3. How the fresh Subject reaches a pooled thread

The observed Subject-propagation window is the ResourceManager startup 
acquisition. The provider runs on a ForkJoinWorkerThread. The fresh Subject 
comes from {{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls 
{{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}. 
That creates a new UGI for the same principal and does not replace the process 
login user.

While {{freshUGI.doAs(...)}} is effective, the provider action calls 
{{fs.addDelegationTokens(...)}}; the blocking HDFS RPC in 
{{CompletableFuture.get()}} enters ForkJoinPool's managed-block path. If no 
idle FJP thread is available and the pool has capacity, {{tryCompensate()}} 
creates additional dispatcher threads during the blocked RPC. This compensation 
does not unblock the RPC; it preserves pool capacity for other work. The 
{{Thread}} constructor for each new thread captures the effective 
AccessControlContext returned by {{AccessController.getContext()}} into that 
thread's {{inheritedAccessControlContext}}; this is a stack-derived snapshot, 
not the entire live Java stack, and it includes the Subject active during 
{{doAs}}. After {{doAs}} returns, current on the provider thread resolves to 
login again, but the newly created threads retain the fresh Subject.

The probe block below combines the observed call chain, Subject identities, and 
thread creation. Here HadoopSecurityContext Subject means the Subject of the 
UGI held by the installed HadoopSecurityContext; it matches the login Subject 
in this probe, but that is not guaranteed in general.

{noformat}
provider thread                     = default-dispatcher-12
freshUGI Subject                    = 817147333
login/HadoopSecurityContext Subject = 1333938290

freshUGI.doAs(...)
  -> fs.addDelegationTokens(...)
  -> HDFS RPC blocked in CompletableFuture.get()
  -> ForkJoinPool.managedBlock()
  -> tryCompensate()
  -> createWorker() [when no idle FJP thread and pool capacity remains]
  -> new dispatcher threads inherit the fresh Subject
  -> later, one of them triggers creation of a cluster-io thread

current Subject before doAs         = 1333938290
current Subject inside doAs         = 817147333
current Subject after doAs          = 1333938290

FJP-created dispatcher threads:
  default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12

later cluster-io thread:
  cluster-io-thread-3, creator=default-dispatcher-15
  current Subject / JM-local receiver target (getCurrentUser()) = 817147333
  login/HadoopSecurityContext Subject = 1333938290
{noformat}

{{fs.addDelegationTokens(...)}} is the provider-side acquisition step: it 
obtains the filesystem delegation token or tokens and stores them in a local 
{{Credentials}} object. The receiver later delivers that object to the selected 
UGI.

{{cluster-io}} is a fixed-size executor; its threads are created lazily on task 
submission. In the probe, an FJP-created dispatcher thread later submits a task 
to this executor, which creates a cluster-io thread that copies the submitter's 
inherited ACC and starts with the fresh Subject. This thread creation is not 
the token refresh itself; it only determines the Subject under which the later 
scheduled refresh runs. A receiver invocation on that thread resolves 
{{getCurrentUser()}} to the fresh Subject and writes credentials there, while 
the login and HadoopSecurityContext UGIs remain unchanged and still resolve to 
the login Subject shown above.

This is a per-thread inherited security context, not a global mutation of 
UserGroupInformation. It is a receiver target-selection mismatch, not a 
client-binding error. Clients created under an unrelated transient Subject 
require separate rebinding or recreation and are out of scope.

h2. Proposed change

This issue fixes token delivery for the observed long-lived-client case, 
primarily checkpoint storage.

* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, select a single target UGI: use the UGI 
held by HadoopSecurityContext when present; otherwise fall back to 
{{UserGroupInformation.getLoginUser()}}. Add refreshed credentials only to that 
UGI. Do not target {{UserGroupInformation.getCurrentUser()}}, which may be a 
pooled thread's inherited Subject.
* Add unit tests for the receiver's context-first and login-fallback target 
selection.

The change makes the delivery target deterministic. It does not rebind a client 
already created under a different Subject.

h2. Scope and boundaries

h3. Out of scope

* Preventing HDFS client initialization or blocking RPCs from occurring inside 
{{freshUGI.doAs(...)}} or the provider's ForkJoinPool thread.
* Closing or rebinding FileSystem instances created under a transient fresh 
Subject; this is a separate client-lifecycle cleanup.
* Delivering credentials to clients created under an unrelated transient 
Subject; those clients require separate rebinding or recreation.
* Changing the provider's {{getCurrentUser()}} usage. 
{{HadoopFSDelegationTokenProvider}} already acquires tokens inside 
{{KerberosLoginProvider.doLoginAndReturnUGI()}}; its remaining 
{{getCurrentUser()}} use is a token-required pre-check and renewer-name lookup. 
No provider change is required for the observed receiver mismatch; it remains a 
separate follow-up if provider-side behavior is observed.
* Double security installation and {{SecurityUtils.install}} idempotency. A 
diagnostic build where the second installation was skipped still failed at the 
24h deadline. The early/duplicate installation adds lifecycle complexity, but 
the probe does not establish it as a cause of the receiver mismatch.

h3. YARN

YARN is not a confirmed reproduction. The YARN samples below were collected 
with a local Java 8-compatible 1.20.5 probe build, not the stock Flink 1.20 
distribution; they do not establish the behavior of the stock JDK11/17 runtime. 
Across repeated controlled application-mode probes, the provider ran on a Pekko 
{{default-dispatcher}} thread and the provider window showed no 
{{tryCompensate}} or {{createWorker}}, so no dispatcher thread was created 
during the fresh-Subject window. The initial actual pool size was not 
consistently larger than the Kubernetes sample: one run started at the same 
actual pool size and still showed no creation. Thus configured parallelism or 
initial pool size alone does not explain the difference. The missing 
discriminator is whether {{CompletableFuture.get()}} entered managed blocking 
and what idle/total state FJP observed. YARN did expand FJP during the separate 
JobGraph-generation path, so it is not immune; with the provider future 
uninstrumented, the trigger remains unestablished.

h3. Versions

Before 1.17, the legacy {{HadoopModule}} path added credentials directly to the 
login user; the {{HadoopDelegationTokenReceiver}} / 
{{UserGroupInformation.getCurrentUser()}} path starts with 1.17. 1.18, 1.19, 
and 1.20 share that receiver behavior. 1.18 is not claimed to be immune.

h2. Testing

* Add receiver unit tests for (a) HadoopSecurityContext UGI precedence with 
login/current not updated, (b) login fallback when no HadoopSecurityContext is 
installed, and (c) the inherited current UGI not targeted.
* Trigger multiple post-startup token refreshes in a long-running job and 
verify which executor runs the provider, whether new ForkJoinPool threads are 
created, and which Subject the receiver's current user resolves to.
* Repeat the YARN provider-window probe while instrumenting the provider's 
{{CompletableFuture}} state; the current YARN sample is summarized under Scope 
and boundaries.
* A deployed application-mode job should survive the 24h token deadline and 
continue checkpointing without SecretManager$InvalidToken. Verify that the 
checkpoint-storage DFSClient receives the new token.
* Run a controlled application-mode probe that verifies the receiver updates 
the UGI selected by the new logic rather than a pooled thread's inherited 
current UGI; cover the TaskManager receiver path separately.

h2. Acceptance criteria

* A token refresh updates the UGI held by HadoopSecurityContext when present, 
otherwise the login UGI, and does not use a pooled thread's inherited current 
UGI; a checkpoint-storage client bound to the selected UGI receives the new 
token.
* A long-running application-mode job survives the 24h HDFS token deadline.


  was:
h2. Summary

Flink periodically obtains HDFS delegation tokens and distributes them through 
HadoopDelegationTokenReceiver. The receiver adds refreshed credentials to 
UserGroupInformation.getCurrentUser() of the thread running the refresh. A 
long-lived Hadoop client, especially a checkpoint-storage DFSClient, may 
instead use the UGI/Subject captured when that client was created. If those 
identities differ, the token refresh succeeds and the receiver reports success, 
but the new credentials are not delivered to the client's UGI. The client 
continues using its startup token and fails at the 24h HDFS token deadline with 
SecretManager$InvalidToken.

In the observed Kubernetes application-mode reproduction, the mismatch becomes 
reachable through ForkJoinPool's managed-block compensation. The provider 
blocks in {{CompletableFuture.get()}} on a ForkJoinWorkerThread; when no idle 
FJP thread is available and the pool has capacity, the compensation path 
creates additional dispatcher threads while {{freshUGI.doAs(...)}} is 
effective, and those threads inherit the fresh Subject. A later {{cluster-io}} 
thread created by one of them inherits the same Subject and can run a token 
receiver under it.

The defect is receiver target selection, not token acquisition or client 
binding. The proposed fix targets the UGI held by HadoopSecurityContext, 
falling back to the login UGI, instead of a pooled thread's current UGI. Here 
"UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means the 
identity it carries; identity comparisons use Subject identity.

h2. Observed failure

This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The 
primary failure was in the checkpoint storage path:

{noformat}
CheckpointCoordinator.initializeCheckpointLocation
  -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
  -> HadoopFileSystem.mkdirs
  -> DFSClient.mkdirs
{noformat}

After the 24h deadline, the checkpoint-storage client reported InvalidToken for 
its startup token even though refresh had obtained a new HDFS token and the 
receiver reported success.

Not every Kubernetes application-mode job is affected. The failure is 
timing-dependent: a long-lived Hadoop client must capture one Subject/UGI 
lifecycle, and a later refresh receiver must run under a different one.

h2. Root cause

The delivery bug is in HadoopDelegationTokenReceiver. It targets 
{{UserGroupInformation.getCurrentUser()}} of the refresh thread. 
{{getCurrentUser()}} resolves from the thread's effective AccessControlContext 
and is not guaranteed to be the UGI held by HadoopSecurityContext or the UGI 
captured by a long-lived checkpoint-storage DFSClient. In the observed failure, 
the checkpoint client's captured UGI had the login/HadoopSecurityContext 
Subject, while the receiver could run under a pooled thread's inherited fresh 
Subject.

h3. Acquisition and refresh phases

Token delivery in the JobManager has three phases. Within each phase, the 
provider and the JM-local receiver run on the same token-manager thread.

* Bootstrap acquisition: {{ClusterEntrypoint.initializeServices()}} calls 
{{DefaultDelegationTokenManager.obtainDelegationTokens()}} on {{main}} before 
BlobServer starts. It updates local receivers for early external filesystem 
access. The receiver runs after {{freshUGI.doAs(...)}} returns, when current is 
login, so this phase is safe.
* ResourceManager startup acquisition: {{ResourceManager.onStart()}} -> 
{{DefaultDelegationTokenManager.start()}} -> {{startTokensUpdate()}} runs on 
{{default-dispatcher}}. This provider call is where the observed ForkJoinPool 
expansion occurs, but the startup receiver itself still runs on 
{{default-dispatcher}} after the provider returns, so it does not target the 
fresh Subject.
* Scheduled refresh: later updates are submitted through {{cluster-io}}. A 
receiver running on a {{cluster-io}} thread that inherited the fresh Subject 
resolves {{getCurrentUser()}} to that Subject and can target the wrong UGI.

TaskManager receivers are invoked by {{TaskExecutor}} token updates, not on 
{{cluster-io}}; the probe in this issue covers the JM-local receiver path only. 
These two startup token acquisitions are separate from the double 
{{SecurityUtils.install}} sequence discussed under Scope and boundaries.

h3. How the fresh Subject reaches a pooled thread

The observed Subject-propagation window is the ResourceManager startup 
acquisition. The provider runs on a ForkJoinWorkerThread. The fresh Subject 
comes from {{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls 
{{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}. 
That creates a new UGI for the same principal and does not replace the process 
login user.

While {{freshUGI.doAs(...)}} is effective, the provider action calls 
{{fs.addDelegationTokens(...)}}; the blocking HDFS RPC in 
{{CompletableFuture.get()}} enters ForkJoinPool's managed-block path. If no 
idle FJP thread is available and the pool has capacity, {{tryCompensate()}} 
creates additional dispatcher threads during the blocked RPC. This compensation 
does not unblock the RPC; it preserves pool capacity for other work. The 
{{Thread}} constructor for each new thread captures the effective 
AccessControlContext returned by {{AccessController.getContext()}} into that 
thread's {{inheritedAccessControlContext}}; this is a stack-derived snapshot, 
not the entire live Java stack, and it includes the Subject active during 
{{doAs}}. After {{doAs}} returns, current on the provider thread resolves to 
login again, but the newly created threads retain the fresh Subject.

The probe block below combines the observed call chain, Subject identities, and 
thread creation. Here HadoopSecurityContext Subject means the Subject of the 
UGI held by the installed HadoopSecurityContext; it matches the login Subject 
in this probe, but that is not guaranteed in general.

{noformat}
provider thread                     = default-dispatcher-12
freshUGI Subject                    = 817147333
login/HadoopSecurityContext Subject = 1333938290

freshUGI.doAs(...)
  -> fs.addDelegationTokens(...)
  -> HDFS RPC blocked in CompletableFuture.get()
  -> ForkJoinPool.managedBlock()
  -> tryCompensate()
  -> createWorker() [when no idle FJP thread and pool capacity remains]
  -> new dispatcher threads inherit the fresh Subject
  -> later, one of them triggers creation of a cluster-io thread

current Subject before doAs         = 1333938290
current Subject inside doAs         = 817147333
current Subject after doAs          = 1333938290

FJP-created dispatcher threads:
  default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12

later cluster-io thread:
  cluster-io-thread-3, creator=default-dispatcher-15
  current Subject / JM-local receiver target (getCurrentUser()) = 817147333
  login/HadoopSecurityContext Subject = 1333938290
{noformat}

{{fs.addDelegationTokens(...)}} is the provider-side acquisition step: it 
obtains the filesystem delegation token or tokens and stores them in a local 
{{Credentials}} object. The receiver later delivers that object to the selected 
UGI.

{{cluster-io}} is a fixed-size executor; its threads are created lazily on task 
submission. In the probe, an FJP-created dispatcher thread later submits a task 
to this executor, which creates a cluster-io thread that copies the submitter's 
inherited ACC and starts with the fresh Subject. This thread creation is not 
the token refresh itself; it only determines the Subject under which the later 
scheduled refresh runs. A receiver invocation on that thread resolves 
{{getCurrentUser()}} to the fresh Subject and writes credentials there, while 
the login and HadoopSecurityContext UGIs remain unchanged and still resolve to 
the login Subject shown above.

This is a per-thread inherited security context, not a global mutation of 
UserGroupInformation. It is a receiver target-selection mismatch, not a 
client-binding error. Clients created under an unrelated transient Subject 
require separate rebinding or recreation and are out of scope.

h2. Proposed change

This issue fixes token delivery for the observed long-lived-client case, 
primarily checkpoint storage.

* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, select a single target UGI: use the UGI 
held by HadoopSecurityContext when present; otherwise fall back to 
{{UserGroupInformation.getLoginUser()}}. Add refreshed credentials only to that 
UGI. Do not target {{UserGroupInformation.getCurrentUser()}}, which may be a 
pooled thread's inherited Subject.
* Add unit tests for the receiver's context-first and login-fallback target 
selection.

The change makes the delivery target deterministic. It does not rebind a client 
already created under a different Subject.

h2. Scope and boundaries

h3. Out of scope

* Preventing HDFS client initialization or blocking RPCs from occurring inside 
{{freshUGI.doAs(...)}} or the provider's ForkJoinPool thread.
* Closing or rebinding FileSystem instances created under a transient fresh 
Subject; this is a separate client-lifecycle cleanup.
* Delivering credentials to clients created under an unrelated transient 
Subject; those clients require separate rebinding or recreation.
* Changing the provider's {{getCurrentUser()}} usage. 
{{HadoopFSDelegationTokenProvider}} already acquires tokens inside 
{{KerberosLoginProvider.doLoginAndReturnUGI()}}; its remaining 
{{getCurrentUser()}} use is a token-required pre-check and renewer-name lookup. 
No provider change is required for the observed receiver mismatch; it remains a 
separate follow-up if provider-side behavior is observed.
* Double security installation and {{SecurityUtils.install}} idempotency. A 
diagnostic build where the second installation was skipped still failed at the 
24h deadline. The early/duplicate installation adds lifecycle complexity, but 
the probe does not establish it as a cause of the receiver mismatch.

h3. YARN

YARN is not a confirmed reproduction. Across repeated controlled 
application-mode probes, the provider ran on a Pekko {{default-dispatcher}} 
thread and the provider window showed no {{tryCompensate}} or {{createWorker}}, 
so no dispatcher thread was created during the fresh-Subject window. The 
initial actual pool size was not consistently larger than the Kubernetes 
sample: one run started at the same actual pool size and still showed no 
creation. Thus configured parallelism or initial pool size alone does not 
explain the difference. The missing discriminator is whether 
{{CompletableFuture.get()}} entered managed blocking and what idle/total state 
FJP observed. YARN did expand FJP during the separate JobGraph-generation path, 
so it is not immune; with the provider future uninstrumented, the trigger 
remains unestablished.

h3. Versions

Before 1.17, the legacy {{HadoopModule}} path added credentials directly to the 
login user; the {{HadoopDelegationTokenReceiver}} / 
{{UserGroupInformation.getCurrentUser()}} path starts with 1.17. 1.18, 1.19, 
and 1.20 share that receiver behavior. 1.18 is not claimed to be immune.

h2. Testing

* Add receiver unit tests for (a) HadoopSecurityContext UGI precedence with 
login/current not updated, (b) login fallback when no HadoopSecurityContext is 
installed, and (c) the inherited current UGI not targeted.
* Trigger multiple post-startup token refreshes in a long-running job and 
verify which executor runs the provider, whether new ForkJoinPool threads are 
created, and which Subject the receiver's current user resolves to.
* Repeat the YARN provider-window probe while instrumenting the provider's 
{{CompletableFuture}} state; the current YARN sample is summarized under Scope 
and boundaries.
* A deployed application-mode job should survive the 24h token deadline and 
continue checkpointing without SecretManager$InvalidToken. Verify that the 
checkpoint-storage DFSClient receives the new token.
* Run a controlled application-mode probe that verifies the receiver updates 
the UGI selected by the new logic rather than a pooled thread's inherited 
current UGI; cover the TaskManager receiver path separately.

h2. Acceptance criteria

* A token refresh updates the UGI held by HadoopSecurityContext when present, 
otherwise the login UGI, and does not use a pooled thread's inherited current 
UGI; a checkpoint-storage client bound to the selected UGI receives the new 
token.
* A long-running application-mode job survives the 24h HDFS token deadline.



> Hadoop delegation token refresh can target a different UGI than Hadoop clients
> ------------------------------------------------------------------------------
>
>                 Key: FLINK-41435
>                 URL: https://issues.apache.org/jira/browse/FLINK-41435
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Configuration
>    Affects Versions: 2.0.0, 1.19.0, 1.20.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0
>         Environment: Flink 1.19.0 and later (1.19.x, 1.20.x, 2.0 through 
> 2.4), plus master (2.5-SNAPSHOT); Kubernetes and standalone application mode 
> with Kerberos and HDFS delegation tokens. The mismatch is more likely when 
> FileSystemJobResultStore and the token manager share the cluster-io executor 
> (FLINK-27204).
>            Reporter: LuChengxiang
>            Priority: Major
>
> h2. Summary
> Flink periodically obtains HDFS delegation tokens and distributes them 
> through HadoopDelegationTokenReceiver. The receiver adds refreshed 
> credentials to UserGroupInformation.getCurrentUser() of the thread running 
> the refresh. A long-lived Hadoop client, especially a checkpoint-storage 
> DFSClient, may instead use the UGI/Subject captured when that client was 
> created. If those identities differ, the token refresh succeeds and the 
> receiver reports success, but the new credentials are not delivered to the 
> client's UGI. The client continues using its startup token and fails at the 
> 24h HDFS token deadline with SecretManager$InvalidToken.
> In the observed Kubernetes application-mode reproduction, the mismatch 
> becomes reachable through ForkJoinPool's managed-block compensation. The 
> provider blocks in {{CompletableFuture.get()}} on a ForkJoinWorkerThread; 
> when no idle FJP thread is available and the pool has capacity, the 
> compensation path creates additional dispatcher threads while 
> {{freshUGI.doAs(...)}} is effective, and those threads inherit the fresh 
> Subject. A later {{cluster-io}} thread created by one of them inherits the 
> same Subject and can run a token receiver under it.
> The defect is receiver target selection, not token acquisition or client 
> binding. The proposed fix targets the UGI held by HadoopSecurityContext, 
> falling back to the login UGI, instead of a pooled thread's current UGI. Here 
> "UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means the 
> identity it carries; identity comparisons use Subject identity.
> h2. Observed failure
> This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
> ZooKeeper-based high availability and HDFS-backed checkpoint storage. The 
> primary failure was in the checkpoint storage path:
> {noformat}
> CheckpointCoordinator.initializeCheckpointLocation
>   -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
>   -> HadoopFileSystem.mkdirs
>   -> DFSClient.mkdirs
> {noformat}
> After the 24h deadline, the checkpoint-storage client reported InvalidToken 
> for its startup token even though refresh had obtained a new HDFS token and 
> the receiver reported success.
> Not every Kubernetes application-mode job is affected. The failure is 
> timing-dependent: a long-lived Hadoop client must capture one Subject/UGI 
> lifecycle, and a later refresh receiver must run under a different one.
> h2. Root cause
> The delivery bug is in HadoopDelegationTokenReceiver. It targets 
> {{UserGroupInformation.getCurrentUser()}} of the refresh thread. 
> {{getCurrentUser()}} resolves from the thread's effective 
> AccessControlContext and is not guaranteed to be the UGI held by 
> HadoopSecurityContext or the UGI captured by a long-lived checkpoint-storage 
> DFSClient. In the observed failure, the checkpoint client's captured UGI had 
> the login/HadoopSecurityContext Subject, while the receiver could run under a 
> pooled thread's inherited fresh Subject.
> h3. Acquisition and refresh phases
> Token delivery in the JobManager has three phases. Within each phase, the 
> provider and the JM-local receiver run on the same token-manager thread.
> * Bootstrap acquisition: {{ClusterEntrypoint.initializeServices()}} calls 
> {{DefaultDelegationTokenManager.obtainDelegationTokens()}} on {{main}} before 
> BlobServer starts. It updates local receivers for early external filesystem 
> access. The receiver runs after {{freshUGI.doAs(...)}} returns, when current 
> is login, so this phase is safe.
> * ResourceManager startup acquisition: {{ResourceManager.onStart()}} -> 
> {{DefaultDelegationTokenManager.start()}} -> {{startTokensUpdate()}} runs on 
> {{default-dispatcher}}. This provider call is where the observed ForkJoinPool 
> expansion occurs, but the startup receiver itself still runs on 
> {{default-dispatcher}} after the provider returns, so it does not target the 
> fresh Subject.
> * Scheduled refresh: later updates are submitted through {{cluster-io}}. A 
> receiver running on a {{cluster-io}} thread that inherited the fresh Subject 
> resolves {{getCurrentUser()}} to that Subject and can target the wrong UGI.
> TaskManager receivers are invoked by {{TaskExecutor}} token updates, not on 
> {{cluster-io}}; the probe in this issue covers the JM-local receiver path 
> only. These two startup token acquisitions are separate from the double 
> {{SecurityUtils.install}} sequence discussed under Scope and boundaries.
> h3. How the fresh Subject reaches a pooled thread
> The observed Subject-propagation window is the ResourceManager startup 
> acquisition. The provider runs on a ForkJoinWorkerThread. The fresh Subject 
> comes from {{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls 
> {{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}. 
> That creates a new UGI for the same principal and does not replace the 
> process login user.
> While {{freshUGI.doAs(...)}} is effective, the provider action calls 
> {{fs.addDelegationTokens(...)}}; the blocking HDFS RPC in 
> {{CompletableFuture.get()}} enters ForkJoinPool's managed-block path. If no 
> idle FJP thread is available and the pool has capacity, {{tryCompensate()}} 
> creates additional dispatcher threads during the blocked RPC. This 
> compensation does not unblock the RPC; it preserves pool capacity for other 
> work. The {{Thread}} constructor for each new thread captures the effective 
> AccessControlContext returned by {{AccessController.getContext()}} into that 
> thread's {{inheritedAccessControlContext}}; this is a stack-derived snapshot, 
> not the entire live Java stack, and it includes the Subject active during 
> {{doAs}}. After {{doAs}} returns, current on the provider thread resolves to 
> login again, but the newly created threads retain the fresh Subject.
> The probe block below combines the observed call chain, Subject identities, 
> and thread creation. Here HadoopSecurityContext Subject means the Subject of 
> the UGI held by the installed HadoopSecurityContext; it matches the login 
> Subject in this probe, but that is not guaranteed in general.
> {noformat}
> provider thread                     = default-dispatcher-12
> freshUGI Subject                    = 817147333
> login/HadoopSecurityContext Subject = 1333938290
> freshUGI.doAs(...)
>   -> fs.addDelegationTokens(...)
>   -> HDFS RPC blocked in CompletableFuture.get()
>   -> ForkJoinPool.managedBlock()
>   -> tryCompensate()
>   -> createWorker() [when no idle FJP thread and pool capacity remains]
>   -> new dispatcher threads inherit the fresh Subject
>   -> later, one of them triggers creation of a cluster-io thread
> current Subject before doAs         = 1333938290
> current Subject inside doAs         = 817147333
> current Subject after doAs          = 1333938290
> FJP-created dispatcher threads:
>   default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12
> later cluster-io thread:
>   cluster-io-thread-3, creator=default-dispatcher-15
>   current Subject / JM-local receiver target (getCurrentUser()) = 817147333
>   login/HadoopSecurityContext Subject = 1333938290
> {noformat}
> {{fs.addDelegationTokens(...)}} is the provider-side acquisition step: it 
> obtains the filesystem delegation token or tokens and stores them in a local 
> {{Credentials}} object. The receiver later delivers that object to the 
> selected UGI.
> {{cluster-io}} is a fixed-size executor; its threads are created lazily on 
> task submission. In the probe, an FJP-created dispatcher thread later submits 
> a task to this executor, which creates a cluster-io thread that copies the 
> submitter's inherited ACC and starts with the fresh Subject. This thread 
> creation is not the token refresh itself; it only determines the Subject 
> under which the later scheduled refresh runs. A receiver invocation on that 
> thread resolves {{getCurrentUser()}} to the fresh Subject and writes 
> credentials there, while the login and HadoopSecurityContext UGIs remain 
> unchanged and still resolve to the login Subject shown above.
> This is a per-thread inherited security context, not a global mutation of 
> UserGroupInformation. It is a receiver target-selection mismatch, not a 
> client-binding error. Clients created under an unrelated transient Subject 
> require separate rebinding or recreation and are out of scope.
> h2. Proposed change
> This issue fixes token delivery for the observed long-lived-client case, 
> primarily checkpoint storage.
> * Add an accessor for the UGI held by HadoopSecurityContext.
> * In HadoopDelegationTokenReceiver, select a single target UGI: use the UGI 
> held by HadoopSecurityContext when present; otherwise fall back to 
> {{UserGroupInformation.getLoginUser()}}. Add refreshed credentials only to 
> that UGI. Do not target {{UserGroupInformation.getCurrentUser()}}, which may 
> be a pooled thread's inherited Subject.
> * Add unit tests for the receiver's context-first and login-fallback target 
> selection.
> The change makes the delivery target deterministic. It does not rebind a 
> client already created under a different Subject.
> h2. Scope and boundaries
> h3. Out of scope
> * Preventing HDFS client initialization or blocking RPCs from occurring 
> inside {{freshUGI.doAs(...)}} or the provider's ForkJoinPool thread.
> * Closing or rebinding FileSystem instances created under a transient fresh 
> Subject; this is a separate client-lifecycle cleanup.
> * Delivering credentials to clients created under an unrelated transient 
> Subject; those clients require separate rebinding or recreation.
> * Changing the provider's {{getCurrentUser()}} usage. 
> {{HadoopFSDelegationTokenProvider}} already acquires tokens inside 
> {{KerberosLoginProvider.doLoginAndReturnUGI()}}; its remaining 
> {{getCurrentUser()}} use is a token-required pre-check and renewer-name 
> lookup. No provider change is required for the observed receiver mismatch; it 
> remains a separate follow-up if provider-side behavior is observed.
> * Double security installation and {{SecurityUtils.install}} idempotency. A 
> diagnostic build where the second installation was skipped still failed at 
> the 24h deadline. The early/duplicate installation adds lifecycle complexity, 
> but the probe does not establish it as a cause of the receiver mismatch.
> h3. YARN
> YARN is not a confirmed reproduction. The YARN samples below were collected 
> with a local Java 8-compatible 1.20.5 probe build, not the stock Flink 1.20 
> distribution; they do not establish the behavior of the stock JDK11/17 
> runtime. Across repeated controlled application-mode probes, the provider ran 
> on a Pekko {{default-dispatcher}} thread and the provider window showed no 
> {{tryCompensate}} or {{createWorker}}, so no dispatcher thread was created 
> during the fresh-Subject window. The initial actual pool size was not 
> consistently larger than the Kubernetes sample: one run started at the same 
> actual pool size and still showed no creation. Thus configured parallelism or 
> initial pool size alone does not explain the difference. The missing 
> discriminator is whether {{CompletableFuture.get()}} entered managed blocking 
> and what idle/total state FJP observed. YARN did expand FJP during the 
> separate JobGraph-generation path, so it is not immune; with the provider 
> future uninstrumented, the trigger remains unestablished.
> h3. Versions
> Before 1.17, the legacy {{HadoopModule}} path added credentials directly to 
> the login user; the {{HadoopDelegationTokenReceiver}} / 
> {{UserGroupInformation.getCurrentUser()}} path starts with 1.17. 1.18, 1.19, 
> and 1.20 share that receiver behavior. 1.18 is not claimed to be immune.
> h2. Testing
> * Add receiver unit tests for (a) HadoopSecurityContext UGI precedence with 
> login/current not updated, (b) login fallback when no HadoopSecurityContext 
> is installed, and (c) the inherited current UGI not targeted.
> * Trigger multiple post-startup token refreshes in a long-running job and 
> verify which executor runs the provider, whether new ForkJoinPool threads are 
> created, and which Subject the receiver's current user resolves to.
> * Repeat the YARN provider-window probe while instrumenting the provider's 
> {{CompletableFuture}} state; the current YARN sample is summarized under 
> Scope and boundaries.
> * A deployed application-mode job should survive the 24h token deadline and 
> continue checkpointing without SecretManager$InvalidToken. Verify that the 
> checkpoint-storage DFSClient receives the new token.
> * Run a controlled application-mode probe that verifies the receiver updates 
> the UGI selected by the new logic rather than a pooled thread's inherited 
> current UGI; cover the TaskManager receiver path separately.
> h2. Acceptance criteria
> * A token refresh updates the UGI held by HadoopSecurityContext when present, 
> otherwise the login UGI, and does not use a pooled thread's inherited current 
> UGI; a checkpoint-storage client bound to the selected UGI receives the new 
> token.
> * A long-running application-mode job survives the 24h HDFS token deadline.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to