[ 
https://issues.apache.org/jira/browse/FLINK-41435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

LuChengxiang updated FLINK-41435:
---------------------------------
    Description: 
h2. Summary

Flink periodically obtains HDFS delegation tokens and distributes them through 
HadoopDelegationTokenReceiver. The receiver adds refreshed credentials to 
UserGroupInformation.getCurrentUser() of the thread running the refresh. A 
long-lived Hadoop client, especially a checkpoint-storage DFSClient, may 
instead use the UGI/Subject captured when that client was created. If those 
identities differ, the refresh succeeds and the receiver reports success, but 
the new credentials are not delivered to the client's UGI. The client keeps 
using its startup token and fails at the 24h HDFS token deadline with 
SecretManager$InvalidToken.

The defect is receiver target selection, not token acquisition or client 
binding. A pooled thread's current UGI can be inherited from a different 
Subject lifecycle than the UGI held by HadoopSecurityContext or the login user. 
The proposed fix targets the UGI held by HadoopSecurityContext, falling back to 
the login UGI, instead of a pooled thread's current UGI.

Here "UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means 
the identity it carries; identity comparisons use Subject identity.

h2. Observed failure

This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The job 
used a Hadoop IPC client with the 3.5.0 CompletableFuture-based wait path. The 
primary failure was in the checkpoint storage path:

{noformat}
CheckpointCoordinator.initializeCheckpointLocation
  -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
  -> HadoopFileSystem.mkdirs
  -> DFSClient.mkdirs
{noformat}

After the 24h deadline, the checkpoint-storage client reported InvalidToken for 
its startup token even though refresh had obtained a new HDFS token and the 
receiver reported success.

Not every application-mode job is affected. The failure is timing-dependent: a 
long-lived Hadoop client must capture one Subject/UGI lifecycle, and a later 
refresh receiver must run under a different one. Whether the provider's 
blocking wait enters FJP managed blocking determines whether a replacement 
worker can be created during the freshUGI.doAs(...) window.

h2. Root cause

HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() of 
the refresh thread. getCurrentUser() resolves from the thread's effective 
AccessControlContext and is not guaranteed to be the UGI held by 
HadoopSecurityContext or the UGI captured by a long-lived checkpoint-storage 
DFSClient.

In the observed failure, the checkpoint client's captured UGI had the 
login/HadoopSecurityContext Subject, while the receiver could run under a 
pooled thread's inherited Subject.

h3. How the refresh thread's current UGI can differ

The provider obtains tokens inside freshUGI.doAs(...). With Apache Hadoop 3.5.0 
and later, Client.getRpcResponse waits on CompletableFuture.get() 
(HADOOP-19235). On a ForkJoinWorkerThread, CompletableFuture.get() enters 
ForkJoinPool.managedBlock(); if no idle worker is available and the pool has 
capacity, ForkJoinPool.tryCompensate() creates a replacement worker while 
freshUGI.doAs(...) is still effective.

The Thread constructor copies the effective AccessControlContext into the new 
thread's inheritedAccessControlContext, so a worker created during that window 
retains the fresh Subject after doAs returns. A later cluster-io worker created 
by one of those workers inherits the same Subject. A receiver running on that 
cluster-io worker then resolves getCurrentUser() to the fresh Subject and 
writes credentials there, while the login and HadoopSecurityContext UGIs remain 
unchanged.

{noformat}
freshUGI.doAs(...)
  -> fs.addDelegationTokens(...)
  -> HDFS RPC in CompletableFuture.get()   [Hadoop 3.5.0+]
  -> ForkJoinPool.managedBlock()
  -> tryCompensate()
  -> createWorker()
  -> new dispatcher worker inherits the fresh Subject
  -> later creates a cluster-io worker
  -> receiver runs with the fresh Subject
{noformat}

In Apache Hadoop 3.3.x and 3.4.x, the same call waits in AsyncGet.Util.wait 
(Object.wait) instead. Object.wait is not an FJP managed block, so the provider 
wait itself does not call tryCompensate; the pool may still create workers for 
unrelated task submission or other managed blocking. This version boundary 
explains why the failure is not reproducible with those clients, but it does 
not make the receiver's getCurrentUser() assumption correct.

h2. Proposed change

This issue fixes token delivery for the observed long-lived-client case, 
primarily checkpoint storage.

* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, select a single target UGI: the UGI held by 
HadoopSecurityContext when present; otherwise fall back to 
UserGroupInformation.getLoginUser(). Add refreshed credentials only to that 
UGI. Do not target UserGroupInformation.getCurrentUser().
* Add unit tests for the receiver's context-first and login-fallback target 
selection.

The change makes the delivery target deterministic. It does not rebind a client 
already created under a different Subject.

h2. Scope and boundaries

h3. Out of scope

* Preventing HDFS client initialization or blocking RPCs from occurring inside 
freshUGI.doAs(...) or the provider's ForkJoinPool thread.
* Closing or rebinding FileSystem instances created under a transient Subject; 
this is a separate client-lifecycle cleanup.
* Changing the provider's getCurrentUser() usage. The provider already acquires 
tokens inside KerberosLoginProvider.doLoginAndReturnUGI(); its remaining 
getCurrentUser() use is a token-required pre-check and renewer-name lookup. A 
provider change is a separate follow-up if provider-side behavior is observed.
* Double security installation and SecurityUtils.install idempotency. A 
diagnostic build where the second installation was skipped still failed at the 
24h deadline.

h3. Version context

Flink: before 1.17, the legacy HadoopModule path added credentials directly to 
the login user; the HadoopDelegationTokenReceiver / getCurrentUser() path 
starts with 1.17. 1.18, 1.19, and 1.20 share that receiver behavior.

Hadoop: Client.getRpcResponse uses CompletableFuture.get() starting with 3.5.0 
(HADOOP-19235). 3.3.x and 3.4.x use AsyncGet.Util.wait. The trigger in this 
issue requires the CompletableFuture-based wait path; the receiver defect is 
independent of the Hadoop version.

h3. Deployment scope

The failure was observed in a Kubernetes application-mode job using the 3.5.0 
IPC behavior. It is not a Kubernetes-specific defect: the trigger is the 
client's wait path and the receiver's executing Subject, not the deployment 
type. Environments using the pre-3.5.0 AsyncGet.Util.wait path do not create 
FJP workers from the provider wait itself.

h2. Testing

* Add receiver unit tests for (a) HadoopSecurityContext UGI precedence, (b) 
login fallback when no HadoopSecurityContext is installed, and (c) the 
inherited current UGI not being targeted.
* A deployed application-mode job using the Hadoop 3.5.0 IPC behavior should 
survive the 24h token deadline and continue checkpointing without 
SecretManager$InvalidToken. Verify that the checkpoint-storage DFSClient 
receives the new token.
* Run a controlled application-mode probe that verifies the receiver updates 
the UGI selected by the new logic rather than a pooled thread's inherited 
current UGI; cover the TaskManager receiver path separately.

h2. Acceptance criteria

* A token refresh updates the UGI held by HadoopSecurityContext when present, 
otherwise the login UGI, and does not use a pooled thread's inherited current 
UGI; a checkpoint-storage client bound to the selected UGI receives the new 
token.
* A long-running application-mode job using the Hadoop 3.5.0 IPC behavior 
survives the 24h HDFS token deadline.


  was:
h2. Summary

Flink periodically obtains HDFS delegation tokens and distributes them through 
HadoopDelegationTokenReceiver. The receiver adds refreshed credentials to 
UserGroupInformation.getCurrentUser() of the thread running the refresh. A 
long-lived Hadoop client, especially a checkpoint-storage DFSClient, may 
instead use the UGI/Subject captured when that client was created. If those 
identities differ, the refresh succeeds and the receiver reports success, but 
the new credentials are not delivered to the client's UGI. The client keeps 
using its startup token and fails at the 24h HDFS token deadline with 
SecretManager$InvalidToken.

The defect is receiver target selection, not token acquisition or client 
binding. A pooled thread's current UGI can be inherited from a different 
Subject lifecycle than the UGI held by HadoopSecurityContext or the login user. 
The proposed fix targets the UGI held by HadoopSecurityContext, falling back to 
the login UGI, instead of a pooled thread's current UGI.

Here "UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means 
the identity it carries; identity comparisons use Subject identity.

h2. Observed failure

This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The job 
used a Hadoop IPC client with the 3.5.0 CompletableFuture-based wait path. The 
primary failure was in the checkpoint storage path:

{noformat}
CheckpointCoordinator.initializeCheckpointLocation
  -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
  -> HadoopFileSystem.mkdirs
  -> DFSClient.mkdirs
{noformat}

After the 24h deadline, the checkpoint-storage client reported InvalidToken for 
its startup token even though refresh had obtained a new HDFS token and the 
receiver reported success.

Not every application-mode job is affected. The failure is timing-dependent: a 
long-lived Hadoop client must capture one Subject/UGI lifecycle, and a later 
refresh receiver must run under a different one. Whether the provider's 
blocking wait enters FJP managed blocking determines whether a replacement 
worker can be created during the freshUGI.doAs(...) window.

h2. Root cause

HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() of 
the refresh thread. getCurrentUser() resolves from the thread's effective 
AccessControlContext and is not guaranteed to be the UGI held by 
HadoopSecurityContext or the UGI captured by a long-lived checkpoint-storage 
DFSClient.

In the observed failure, the checkpoint client's captured UGI had the 
login/HadoopSecurityContext Subject, while the receiver could run under a 
pooled thread's inherited Subject.

h3. How the refresh thread's current UGI can differ

The provider obtains tokens inside freshUGI.doAs(...). With Apache Hadoop 3.5.0 
and later, Client.getRpcResponse waits on CompletableFuture.get() 
(HADOOP-19235). On a ForkJoinWorkerThread, CompletableFuture.get() enters 
ForkJoinPool.managedBlock(); if no idle worker is available and the pool has 
capacity, ForkJoinPool.tryCompensate() creates a replacement worker while 
freshUGI.doAs(...) is still effective.

The Thread constructor copies the effective AccessControlContext into the new 
thread's inheritedAccessControlContext, so a worker created during that window 
retains the fresh Subject after doAs returns. A later cluster-io worker created 
by one of those workers inherits the same Subject. A receiver running on that 
cluster-io worker then resolves getCurrentUser() to the fresh Subject and 
writes credentials there, while the login and HadoopSecurityContext UGIs remain 
unchanged.

{noformat}
freshUGI.doAs(...)
  -> fs.addDelegationTokens(...)
  -> HDFS RPC in CompletableFuture.get()   [Hadoop 3.5.0+]
  -> ForkJoinPool.managedBlock()
  -> tryCompensate()
  -> createWorker()
  -> new dispatcher worker inherits the fresh Subject
  -> later creates a cluster-io worker
  -> receiver runs with the fresh Subject
{noformat}

In Apache Hadoop 3.3.x and 3.4.x, the same call waits in AsyncGet.Util.wait 
(Object.wait) instead. Object.wait is not an FJP managed block, so the pool 
does not create a replacement worker; the provider thread blocks without 
propagating the fresh Subject to a new thread. This version boundary explains 
why the failure is not reproducible with those clients, but it does not make 
the receiver's getCurrentUser() assumption correct.

h2. Proposed change

This issue fixes token delivery for the observed long-lived-client case, 
primarily checkpoint storage.

* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, select a single target UGI: the UGI held by 
HadoopSecurityContext when present; otherwise fall back to 
UserGroupInformation.getLoginUser(). Add refreshed credentials only to that 
UGI. Do not target UserGroupInformation.getCurrentUser().
* Add unit tests for the receiver's context-first and login-fallback target 
selection.

The change makes the delivery target deterministic. It does not rebind a client 
already created under a different Subject.

h2. Scope and boundaries

h3. Out of scope

* Preventing HDFS client initialization or blocking RPCs from occurring inside 
freshUGI.doAs(...) or the provider's ForkJoinPool thread.
* Closing or rebinding FileSystem instances created under a transient Subject; 
this is a separate client-lifecycle cleanup.
* Changing the provider's getCurrentUser() usage. The provider already acquires 
tokens inside KerberosLoginProvider.doLoginAndReturnUGI(); its remaining 
getCurrentUser() use is a token-required pre-check and renewer-name lookup. A 
provider change is a separate follow-up if provider-side behavior is observed.
* Double security installation and SecurityUtils.install idempotency. A 
diagnostic build where the second installation was skipped still failed at the 
24h deadline.

h3. Version context

Flink: before 1.17, the legacy HadoopModule path added credentials directly to 
the login user; the HadoopDelegationTokenReceiver / getCurrentUser() path 
starts with 1.17. 1.18, 1.19, and 1.20 share that receiver behavior.

Hadoop: Client.getRpcResponse uses CompletableFuture.get() starting with 3.5.0 
(HADOOP-19235). 3.3.x and 3.4.x use AsyncGet.Util.wait. The trigger in this 
issue requires the CompletableFuture-based wait path; the receiver defect is 
independent of the Hadoop version.

h3. Deployment scope

The failure was observed in a Kubernetes application-mode job using the 3.5.0 
IPC behavior. It is not a Kubernetes-specific defect: the trigger is the 
client's wait path and the receiver's executing Subject, not the deployment 
type. Environments using the pre-3.5.0 AsyncGet.Util.wait path do not create 
FJP workers from the provider wait itself.

h2. Testing

* Add receiver unit tests for (a) HadoopSecurityContext UGI precedence, (b) 
login fallback when no HadoopSecurityContext is installed, and (c) the 
inherited current UGI not being targeted.
* A deployed application-mode job using the Hadoop 3.5.0 IPC behavior should 
survive the 24h token deadline and continue checkpointing without 
SecretManager$InvalidToken. Verify that the checkpoint-storage DFSClient 
receives the new token.
* Run a controlled application-mode probe that verifies the receiver updates 
the UGI selected by the new logic rather than a pooled thread's inherited 
current UGI; cover the TaskManager receiver path separately.

h2. Acceptance criteria

* A token refresh updates the UGI held by HadoopSecurityContext when present, 
otherwise the login UGI, and does not use a pooled thread's inherited current 
UGI; a checkpoint-storage client bound to the selected UGI receives the new 
token.
* A long-running application-mode job using the Hadoop 3.5.0 IPC behavior 
survives the 24h HDFS token deadline.



> Hadoop delegation token refresh can target a different UGI than Hadoop clients
> ------------------------------------------------------------------------------
>
>                 Key: FLINK-41435
>                 URL: https://issues.apache.org/jira/browse/FLINK-41435
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Configuration
>    Affects Versions: 2.0.0, 1.19.0, 1.20.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0
>         Environment: Flink 1.19.0 and later (1.19.x, 1.20.x, 2.0 through 
> 2.4), plus master (2.5-SNAPSHOT); Kubernetes and standalone application mode 
> with Kerberos and HDFS delegation tokens. The mismatch is more likely when 
> FileSystemJobResultStore and the token manager share the cluster-io executor 
> (FLINK-27204).
>            Reporter: LuChengxiang
>            Priority: Major
>
> h2. Summary
> Flink periodically obtains HDFS delegation tokens and distributes them 
> through HadoopDelegationTokenReceiver. The receiver adds refreshed 
> credentials to UserGroupInformation.getCurrentUser() of the thread running 
> the refresh. A long-lived Hadoop client, especially a checkpoint-storage 
> DFSClient, may instead use the UGI/Subject captured when that client was 
> created. If those identities differ, the refresh succeeds and the receiver 
> reports success, but the new credentials are not delivered to the client's 
> UGI. The client keeps using its startup token and fails at the 24h HDFS token 
> deadline with SecretManager$InvalidToken.
> The defect is receiver target selection, not token acquisition or client 
> binding. A pooled thread's current UGI can be inherited from a different 
> Subject lifecycle than the UGI held by HadoopSecurityContext or the login 
> user. The proposed fix targets the UGI held by HadoopSecurityContext, falling 
> back to the login UGI, instead of a pooled thread's current UGI.
> Here "UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means 
> the identity it carries; identity comparisons use Subject identity.
> h2. Observed failure
> This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
> ZooKeeper-based high availability and HDFS-backed checkpoint storage. The job 
> used a Hadoop IPC client with the 3.5.0 CompletableFuture-based wait path. 
> The primary failure was in the checkpoint storage path:
> {noformat}
> CheckpointCoordinator.initializeCheckpointLocation
>   -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
>   -> HadoopFileSystem.mkdirs
>   -> DFSClient.mkdirs
> {noformat}
> After the 24h deadline, the checkpoint-storage client reported InvalidToken 
> for its startup token even though refresh had obtained a new HDFS token and 
> the receiver reported success.
> Not every application-mode job is affected. The failure is timing-dependent: 
> a long-lived Hadoop client must capture one Subject/UGI lifecycle, and a 
> later refresh receiver must run under a different one. Whether the provider's 
> blocking wait enters FJP managed blocking determines whether a replacement 
> worker can be created during the freshUGI.doAs(...) window.
> h2. Root cause
> HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() 
> of the refresh thread. getCurrentUser() resolves from the thread's effective 
> AccessControlContext and is not guaranteed to be the UGI held by 
> HadoopSecurityContext or the UGI captured by a long-lived checkpoint-storage 
> DFSClient.
> In the observed failure, the checkpoint client's captured UGI had the 
> login/HadoopSecurityContext Subject, while the receiver could run under a 
> pooled thread's inherited Subject.
> h3. How the refresh thread's current UGI can differ
> The provider obtains tokens inside freshUGI.doAs(...). With Apache Hadoop 
> 3.5.0 and later, Client.getRpcResponse waits on CompletableFuture.get() 
> (HADOOP-19235). On a ForkJoinWorkerThread, CompletableFuture.get() enters 
> ForkJoinPool.managedBlock(); if no idle worker is available and the pool has 
> capacity, ForkJoinPool.tryCompensate() creates a replacement worker while 
> freshUGI.doAs(...) is still effective.
> The Thread constructor copies the effective AccessControlContext into the new 
> thread's inheritedAccessControlContext, so a worker created during that 
> window retains the fresh Subject after doAs returns. A later cluster-io 
> worker created by one of those workers inherits the same Subject. A receiver 
> running on that cluster-io worker then resolves getCurrentUser() to the fresh 
> Subject and writes credentials there, while the login and 
> HadoopSecurityContext UGIs remain unchanged.
> {noformat}
> freshUGI.doAs(...)
>   -> fs.addDelegationTokens(...)
>   -> HDFS RPC in CompletableFuture.get()   [Hadoop 3.5.0+]
>   -> ForkJoinPool.managedBlock()
>   -> tryCompensate()
>   -> createWorker()
>   -> new dispatcher worker inherits the fresh Subject
>   -> later creates a cluster-io worker
>   -> receiver runs with the fresh Subject
> {noformat}
> In Apache Hadoop 3.3.x and 3.4.x, the same call waits in AsyncGet.Util.wait 
> (Object.wait) instead. Object.wait is not an FJP managed block, so the 
> provider wait itself does not call tryCompensate; the pool may still create 
> workers for unrelated task submission or other managed blocking. This version 
> boundary explains why the failure is not reproducible with those clients, but 
> it does not make the receiver's getCurrentUser() assumption correct.
> h2. Proposed change
> This issue fixes token delivery for the observed long-lived-client case, 
> primarily checkpoint storage.
> * Add an accessor for the UGI held by HadoopSecurityContext.
> * In HadoopDelegationTokenReceiver, select a single target UGI: the UGI held 
> by HadoopSecurityContext when present; otherwise fall back to 
> UserGroupInformation.getLoginUser(). Add refreshed credentials only to that 
> UGI. Do not target UserGroupInformation.getCurrentUser().
> * Add unit tests for the receiver's context-first and login-fallback target 
> selection.
> The change makes the delivery target deterministic. It does not rebind a 
> client already created under a different Subject.
> h2. Scope and boundaries
> h3. Out of scope
> * Preventing HDFS client initialization or blocking RPCs from occurring 
> inside freshUGI.doAs(...) or the provider's ForkJoinPool thread.
> * Closing or rebinding FileSystem instances created under a transient 
> Subject; this is a separate client-lifecycle cleanup.
> * Changing the provider's getCurrentUser() usage. The provider already 
> acquires tokens inside KerberosLoginProvider.doLoginAndReturnUGI(); its 
> remaining getCurrentUser() use is a token-required pre-check and renewer-name 
> lookup. A provider change is a separate follow-up if provider-side behavior 
> is observed.
> * Double security installation and SecurityUtils.install idempotency. A 
> diagnostic build where the second installation was skipped still failed at 
> the 24h deadline.
> h3. Version context
> Flink: before 1.17, the legacy HadoopModule path added credentials directly 
> to the login user; the HadoopDelegationTokenReceiver / getCurrentUser() path 
> starts with 1.17. 1.18, 1.19, and 1.20 share that receiver behavior.
> Hadoop: Client.getRpcResponse uses CompletableFuture.get() starting with 
> 3.5.0 (HADOOP-19235). 3.3.x and 3.4.x use AsyncGet.Util.wait. The trigger in 
> this issue requires the CompletableFuture-based wait path; the receiver 
> defect is independent of the Hadoop version.
> h3. Deployment scope
> The failure was observed in a Kubernetes application-mode job using the 3.5.0 
> IPC behavior. It is not a Kubernetes-specific defect: the trigger is the 
> client's wait path and the receiver's executing Subject, not the deployment 
> type. Environments using the pre-3.5.0 AsyncGet.Util.wait path do not create 
> FJP workers from the provider wait itself.
> h2. Testing
> * Add receiver unit tests for (a) HadoopSecurityContext UGI precedence, (b) 
> login fallback when no HadoopSecurityContext is installed, and (c) the 
> inherited current UGI not being targeted.
> * A deployed application-mode job using the Hadoop 3.5.0 IPC behavior should 
> survive the 24h token deadline and continue checkpointing without 
> SecretManager$InvalidToken. Verify that the checkpoint-storage DFSClient 
> receives the new token.
> * Run a controlled application-mode probe that verifies the receiver updates 
> the UGI selected by the new logic rather than a pooled thread's inherited 
> current UGI; cover the TaskManager receiver path separately.
> h2. Acceptance criteria
> * A token refresh updates the UGI held by HadoopSecurityContext when present, 
> otherwise the login UGI, and does not use a pooled thread's inherited current 
> UGI; a checkpoint-storage client bound to the selected UGI receives the new 
> token.
> * A long-running application-mode job using the Hadoop 3.5.0 IPC behavior 
> survives the 24h HDFS token deadline.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to