[ 
https://issues.apache.org/jira/browse/FLINK-41435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

LuChengxiang updated FLINK-41435:
---------------------------------
    Description: 
h2. Summary

Flink periodically obtains HDFS delegation tokens and distributes them through 
HadoopDelegationTokenReceiver. The receiver currently adds new credentials to 
UserGroupInformation.getCurrentUser() on the thread that performs the refresh. 
Long-lived Hadoop clients, especially checkpoint storage, can capture a 
UGI/Subject when their DFSClient is created and use that captured identity for 
later RPC connections.

If the refresh path resolves to a different current UGI from the one captured 
by a long-lived client, the client can continue using its startup token even 
though a new token was obtained successfully. The failure appears at the 24h 
HDFS token deadline as SecretManager$InvalidToken.

h2. Observed failure

This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The 
primary failure was in the checkpoint storage path:

{noformat}
CheckpointCoordinator.initializeCheckpointLocation
  -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
  -> HadoopFileSystem.mkdirs
  -> DFSClient.mkdirs
{noformat}

The checkpoint storage client reported InvalidToken for the startup token after 
the refresh had created a new HDFS token and the receiver reported success. The 
same JVM also showed a FileSystemJobResultStore stack using the expired token 
during cleanup. The job-result store is a secondary long-lived client with the 
same failure mode, not the primary trigger.

Not every Kubernetes application-mode job is affected. The failure is 
timing-dependent: it requires a long-lived Hadoop client to have captured one 
Subject/UGI lifecycle while the refresh path later resolves to a different one.

Installation idempotency is out of scope. A diagnostic build that skipped the 
second security installation still failed at the 24h HDFS token deadline.

h2. Root cause

HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() of 
the thread executing the refresh. That current UGI is resolved from the 
thread's effective AccessControlContext and is not guaranteed to be the UGI 
held by HadoopSecurityContext or the UGI captured by a long-lived 
checkpoint-storage or job-result-store DFSClient.

The controlled Kubernetes probe observed this path:

{noformat}
provider thread                 = default-dispatcher-12
freshUGI Subject                = 817147333
login/security context Subject  = 1333938290

current Subject before doAs    = 1333938290
current Subject inside doAs    = 817147333
current Subject after doAs     = 1333938290

FJP workers created during the provider's blocked RPC:
  default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12

later cluster-io worker:
  cluster-io-thread-3, creator=default-dispatcher-15, subjectId=817147333
{noformat}

The fresh Subject is not created by HadoopSecurityContext or by the dispatcher. 
It belongs to the UGI returned by 
{{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls 
{{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}. 
That call performs a Kerberos login and returns a newly constructed UGI for the 
same principal; it does not replace the process login user.

During {{freshUGI.doAs(...)}}, {{fs.addDelegationTokens(...)}} performed an 
HDFS delegation-token RPC and blocked in {{CompletableFuture.get()}}. Because 
the caller was a ForkJoinWorkerThread, ForkJoinPool entered its managed-block 
compensation path and created {{default-dispatcher-13}}, 
{{default-dispatcher-14}}, and {{default-dispatcher-15}} while the fresh 
Subject was active. The Thread constructors copied the creator's effective 
AccessControlContext, including the fresh Subject. After {{doAs}} returned, 
{{getCurrentUser()}} on the creator resolved to the login Subject again, but 
the newly created workers retained the inherited fresh Subject.

Later, {{default-dispatcher-15}} created {{cluster-io-thread-3}}; the probe 
recorded {{subjectId=817147333}} for that worker while {{getLoginUser()}} and 
{{HadoopSecurityContext}} still referred to the login Subject {{1333938290}}. A 
receiver task scheduled on such a worker can therefore add credentials to the 
fresh Subject rather than the UGI captured by the long-lived client. This is a 
per-thread inherited security context, not a global mutation of 
{{UserGroupInformation}} or the process login user.

The provider stack identifies this invocation as {{ResourceManager.onStart}} -> 
{{DefaultDelegationTokenManager.start()}} -> {{startTokensUpdate()}}, i.e. the 
initial token acquisition. The later scheduled refresh is submitted through the 
cluster-io executor. The startup FJP expansion is observed; the same mechanism 
is possible for any provider {{doAs}} on a ForkJoinWorkerThread with a blocking 
HDFS RPC, but that later path has not been directly reproduced in this probe.

h2. YARN status and scope

In a controlled YARN application-mode probe, the provider also ran on a Pekko 
default-dispatcher worker. The provider window showed {{poolSize=3}} / 
{{active=2}} before and after, with no {{tryCompensate}} or {{createWorker}}; 
the current Subject after the provider returned was the login Subject, while 
the provider's internal {{freshUGI}} Subject was a different value. YARN did 
show ForkJoinPool expansion during the separate JobGraph-generation startup 
path, so this is not evidence that YARN is immune.

This is one YARN sample; the provider's {{CompletableFuture}} state was not 
instrumented, so the YARN trigger is not established. YARN is not listed as a 
confirmed reproduction, but it is not claimed to be exempt.

h2. Proposed change

This issue addresses the case where a token is obtained successfully but the 
refreshed credentials do not reach long-lived Hadoop clients, primarily 
checkpoint storage.

* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, add refreshed credentials to the 
security-context UGI, the current UGI, and the login UGI. Deduplicate the 
target UGI references by object identity; this prevents duplicate 
{{addCredentials}} calls for the same wrapper, but does not merge distinct UGI 
wrappers that reference the same Subject. The security-context UGI is the 
primary target; the other two preserve non-secured deployments and previous 
behavior.
* Add tests covering (a) a refresh thread whose current user differs from the 
login user, and (b) a security-context UGI distinct from the current/login 
user; verify that all applicable targets receive the credentials.

Out of scope for this issue:

* Preventing HDFS client initialization or blocking RPCs from occurring inside 
{{freshUGI.doAs(...)}} or the provider's ForkJoinPool worker.
* Closing or rebinding FileSystem instances created under a transient fresh 
Subject; this is a separate client-lifecycle cleanup.
* SecurityUtils.install idempotency.

The proposed receiver change does not rely solely on the refresh thread's 
inherited Subject. It does not by itself guarantee that a client already 
created under a different Subject is rebound. That remains a separate lifecycle 
concern.

h2. Testing

* A controlled Kubernetes application-mode probe recorded the chain from 
{{fs.addDelegationTokens()}} through Hadoop ipc {{Client.getRpcResponse()}}, 
{{CompletableFuture.get()}}, {{ForkJoinPool.managedBlock()}}, 
{{tryCompensate()}}, and Pekko {{ThreadFactory.newThread()}}. The creator and 
the new dispatcher workers had the same fresh Subject identity.
* The same probe recorded {{cluster-io-thread-3}} created by 
{{default-dispatcher-15}} with {{subjectId=817147333}}, while the login and 
security-context Subjects remained {{1333938290}}.
* Trigger multiple post-startup token refreshes in a long-running job and 
verify which executor runs the provider, whether new ForkJoinPool workers are 
created, and which Subject the receiver's current user resolves to.
* The controlled YARN application-mode probe showed no {{tryCompensate}} or 
{{createWorker}} in the provider window. The separate JobGraph-generation 
startup path did create FJP workers, confirming that YARN can enter the same 
FJP mechanism under different startup work; the provider's 
{{CompletableFuture}} blocking state remains unverified.
* A deployed application-mode job should survive the 24h token deadline and 
continue checkpointing without SecretManager$InvalidToken. Verify that the 
checkpoint-storage DFSClient receives the new token, and verify the 
job-result-store client separately because it can capture a different UGI.

h2. Acceptance criteria

* A token refresh updates the UGI used by checkpoint storage and other 
long-lived Hadoop clients.
* Refresh delivers credentials to the security-context UGI regardless of which 
Subject a pooled dispatcher or cluster-io worker inherited.
* A long-running application-mode job survives the 24h HDFS token deadline.


  was:
h2. Background

Flink periodically obtains HDFS delegation tokens and distributes them through 
HadoopDelegationTokenReceiver. The receiver currently adds new credentials to 
UserGroupInformation.getCurrentUser() on the thread that performs the refresh. 
Long-lived Hadoop clients, especially checkpoint storage and 
FileSystemJobResultStore, capture a UGI/Subject when their DFSClient is created 
and use that captured identity for later RPC connections.

If the refresh path resolves to a different current UGI from the one captured 
by a long-lived client, the client can continue using the startup token even 
though a new token was obtained successfully. The failure appears at the 24h 
HDFS token deadline as SecretManager$InvalidToken.

h2. Observed failure

This was observed in a Kubernetes application-mode job using ZooKeeper-based 
high availability and HDFS-backed checkpoint storage. The primary failure was 
in the checkpoint storage path:

{noformat}
CheckpointCoordinator.initializeCheckpointLocation
  -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
  -> HadoopFileSystem.mkdirs
  -> DFSClient.mkdirs
{noformat}

The checkpoint storage client reported InvalidToken for the startup token after 
the refresh had created a new HDFS token and the receiver reported success. The 
same JVM also showed a FileSystemJobResultStore stack using the expired token 
during cleanup. The job-result store is therefore a secondary long-lived client 
with the same failure mode, not the primary trigger.

Not every Kubernetes application-mode job is affected. The failure is 
timing-dependent: it requires a long-lived Hadoop client to have captured one 
Subject/UGI lifecycle while the refresh path later resolves to a different one.

The double security installation is not the root cause: a diagnostic build that 
skipped the second security installation still failed at the 24h HDFS token 
deadline. This issue therefore focuses on the receiver's `current` UGI 
behavior; installation idempotency remains out of scope.

h2. New finding: how a fresh Subject reaches a pooled Pekko worker

A controlled Kubernetes application-mode probe instrumented the token provider, 
the HDFS RPC path, and Pekko's dispatcher thread factory. The observed chain 
was:

{noformat}
HadoopFSDelegationTokenProvider.obtainDelegationTokens()
  -> freshUGI.doAs(...)
  -> fs.addDelegationTokens(...)
  -> HDFS delegation-token RPC
  -> Client.getRpcResponse()
  -> CompletableFuture.get()
  -> ForkJoinPool.managedBlock()
  -> tryCompensate()
  -> createWorker()
  -> Pekko ThreadFactory.newThread()
{noformat}

This happened while a Pekko default-dispatcher worker was executing 
freshUGI.doAs(...). The HDFS RPC was waiting on a CompletableFuture. Because 
the caller was a ForkJoinWorkerThread, JDK ForkJoinPool entered its 
managed-block compensation path and created additional dispatcher workers while 
the fresh Subject was active.

h3. Origin of the fresh Subject

The fresh Subject is not created by HadoopSecurityContext or by the dispatcher. 
It belongs to the UGI returned by 
{{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls 
{{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}} 
when a principal is configured. That call performs a Kerberos login and returns 
a newly constructed UGI for the same principal; it does not replace the process 
login user. In the probe, the provider observed:

{noformat}
provider thread                 = default-dispatcher-12
freshUGI Subject                = 817147333
login/security context Subject  = 1333938290

current Subject before doAs    = 1333938290
current Subject inside doAs    = 817147333
current Subject after doAs     = 1333938290

FJP workers created during the provider's blocked RPC:
  default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12

later cluster-io worker:
  cluster-io-thread-3, creator=default-dispatcher-15, subjectId=817147333
{noformat}

In this probe the provider was invoked by {{ResourceManager.onStart}} -> 
{{DefaultDelegationTokenManager.start()}} -> {{startTokensUpdate()}}, i.e. the 
initial token acquisition, not the later scheduled refresh.

While the provider was executing {{freshUGI.doAs(...)}}, 
{{fs.addDelegationTokens(...)}} performed an HDFS delegation-token RPC and 
blocked in {{CompletableFuture.get()}}. Because the caller was a 
ForkJoinWorkerThread, ForkJoinPool entered its managed-block compensation path 
and created {{default-dispatcher-13}}, {{default-dispatcher-14}}, and 
{{default-dispatcher-15}} with {{default-dispatcher-12}} as their creator. The 
Thread constructors copied the creator's effective AccessControlContext, 
including the fresh Subject. After {{doAs}} returned, {{getCurrentUser()}} on 
the creator resolved to the login Subject again, but the newly created workers 
retained the inherited fresh Subject.

Later, {{default-dispatcher-15}} created {{cluster-io-thread-3}}. The probe 
recorded {{subjectId=817147333}} for that worker, while {{getLoginUser()}} and 
{{HadoopSecurityContext}} still referred to the login Subject {{1333938290}}. 
This is the concrete propagation path from the provider's {{freshUGI}} to a 
worker that later resolves {{UserGroupInformation.getCurrentUser()}} to the 
fresh Subject.

This is a per-thread inherited security context, not a global mutation of 
{{UserGroupInformation}} or the process login user.

{{UserGroupInformation.getCurrentUser()}} returns the current user including 
any {{doAs}} scope on the current stack. {{Subject.doAs}} constructs a 
Subject-bound {{AccessControlContext}} and makes it effective through 
{{AccessController.doPrivileged}}; during the action, 
{{AccessController.getContext()}} observes that inner context. A {{Thread}} 
created inside this window stores the context returned by 
{{AccessController.getContext()}} in its {{inheritedAccessControlContext}}, so 
the pooled worker can keep the inner fresh Subject after {{doAs}} returns.

The probe observed this HDFS RPC path while the provider was running on a 
ForkJoinPool worker. The resulting dispatcher workers inherited the fresh 
Subject, so a later UserGroupInformation.getCurrentUser() call on a pooled 
worker can resolve to that fresh Subject rather than the security-context or 
login Subject. This is the `current` UGI used by the receiver.

This is not a startup-only condition: the token provider executes on each 
refresh, and a later blocking HDFS RPC on a ForkJoinPool worker while 
freshUGI.doAs() is active can create another worker inheriting the fresh 
Subject. The startup path is where the expansion was observed; the inherited 
Subject persists on the worker and can be used by later refresh or receiver 
tasks.

h2. YARN status

In a controlled YARN application-mode probe, refresh-time current and login 
Subject identities were equal. The provider did run on a Pekko dispatcher 
worker, but all provider invocations reported no newly created Pekko threads. 
The normal YARN application-mode startup path tested so far did not reproduce 
the fresh-Subject worker creation observed on Kubernetes.

This absence of new workers is not proof that YARN never enters the 
managedBlock/createWorker path. A YARN AM-level full-stack capture has not been 
completed, so the platform-specific trigger is not yet established. YARN is 
therefore not listed as a confirmed reproduction, but it is also not claimed to 
be immune. The code path is available on YARN if a token-provider call creates 
a new HDFS client and blocks on a ForkJoinWorkerThread; that blocking-client 
path still needs to be confirmed there.

h2. Version and code-path notes

The 1.18 path had synchronous FileSystemJobResultStore I/O, so the 
worker/Subject interaction was not triggered in the deployments observed. 1.15 
did not have the unified delegation-token manager path. These version 
differences may explain the observed frequency; they do not prove that those 
versions are immune.

FLINK-27204 (1.19+) moved FileSystemJobResultStore I/O onto the shared 
cluster-io executor. DefaultDelegationTokenManager uses the same executor for 
periodic token refresh. The provider and receiver are invoked on the refresh 
path; the security-context UGI, login UGI, and current UGI can diverge.

h2. Root cause

HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() of 
the thread executing the refresh. That current UGI is resolved from the 
thread's effective AccessControlContext and is not guaranteed to be the UGI 
held by HadoopSecurityContext or the UGI captured by a long-lived 
checkpoint-storage or job-result-store DFSClient.

In the reproduced Kubernetes path, the freshUGI.doAs() window caused a new 
Pekko dispatcher worker to inherit the fresh Subject. Consequently, the 
receiver could execute with a current UGI that differed from the 
security-context and login UGIs. The second security installation is not 
required for this mismatch, and install idempotency therefore does not address 
it.

h2. Proposed change

This issue addresses the case where a token is obtained successfully but the 
refreshed credentials do not reach long-lived Hadoop clients, primarily 
checkpoint storage.

* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, add refreshed credentials to the 
security-context UGI, the current UGI, and the login UGI, deduplicated by 
object identity. The security-context UGI is the primary target; the other two 
preserve non-secured deployments and previous behavior.
* Add tests covering (a) a refresh thread whose current user differs from the 
login user, and (b) a security-context UGI distinct from the current/login 
user; verify that all applicable targets receive the credentials.

Out of scope for this issue:

* Preventing HDFS client initialization or blocking RPCs from occurring inside 
freshUGI.doAs() or the provider's ForkJoinPool worker.
* Closing or rebinding FileSystem instances created under a transient fresh 
Subject; this is a separate client-lifecycle cleanup.
* SecurityUtils.install idempotency.

The proposed receiver change does not rely solely on the refresh thread's 
inherited Subject. It does not by itself guarantee that a client already 
created under a different Subject is rebound. That remains a separate lifecycle 
concern.

h2. Testing

* A controlled Kubernetes application-mode probe recorded the full chain from 
fs.addDelegationTokens() through Hadoop ipc Client.getRpcResponse(), 
CompletableFuture.get(), ForkJoinPool.managedBlock(), tryCompensate(), and 
Pekko ThreadFactory.newThread(). The creator and the new dispatcher workers had 
the same fresh Subject identity.
* A later receiver invocation on a cluster-io worker observed that inherited 
fresh Subject while the login and security-context Subjects remained the 
original login Subject.
* Trigger multiple post-startup token refreshes in a long-running job and 
verify which executor runs the provider, whether new ForkJoinPool workers are 
created, and which Subject the receiver's current user resolves to.
* A JVM probe confirmed that a fixed thread pool created under Subject A but 
first submitted under Subject B runs subsequent tasks under Subject B.
* A controlled Kubernetes application-mode probe instrumented 
ExecutorThreadFactory. The first cluster-io worker was created by 
DefaultLeaderElectionService-leadershipOperationExecutor-thread-1, and the 
first jobmanager-io worker by a cluster-io worker; both inherited the same 
Kerberos Subject at creation time. This is a supporting inheritance check, not 
evidence for the ForkJoinPool compensation path.
* Receiver tests as described above.
* The controlled YARN application-mode probe observed no new Pekko dispatcher 
workers in the tested normal startup path. A YARN AM-level full-stack capture 
remains to be completed.
* A deployed application-mode job should survive the 24h token deadline and 
continue checkpointing without SecretManager$InvalidToken. Verify that the 
checkpoint-storage DFSClient receives the new token, and verify the 
job-result-store client separately because it can capture a different UGI.

h2. Acceptance criteria

* A token refresh updates the UGI used by checkpoint storage and other 
long-lived Hadoop clients.
* Refresh delivers credentials to the security-context UGI regardless of which 
Subject a pooled dispatcher or cluster-io worker inherited.
* A long-running application-mode job survives the 24h HDFS token deadline.



> Hadoop delegation token refresh can target a different UGI than Hadoop clients
> ------------------------------------------------------------------------------
>
>                 Key: FLINK-41435
>                 URL: https://issues.apache.org/jira/browse/FLINK-41435
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Configuration
>    Affects Versions: 2.0.0, 1.19.0, 1.20.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0
>         Environment: Flink 1.19.0 and later (1.19.x, 1.20.x, 2.0 through 
> 2.4), plus master (2.5-SNAPSHOT); Kubernetes and standalone application mode 
> with Kerberos and HDFS delegation tokens. The mismatch is more likely when 
> FileSystemJobResultStore and the token manager share the cluster-io executor 
> (FLINK-27204).
>            Reporter: LuChengxiang
>            Priority: Major
>
> h2. Summary
> Flink periodically obtains HDFS delegation tokens and distributes them 
> through HadoopDelegationTokenReceiver. The receiver currently adds new 
> credentials to UserGroupInformation.getCurrentUser() on the thread that 
> performs the refresh. Long-lived Hadoop clients, especially checkpoint 
> storage, can capture a UGI/Subject when their DFSClient is created and use 
> that captured identity for later RPC connections.
> If the refresh path resolves to a different current UGI from the one captured 
> by a long-lived client, the client can continue using its startup token even 
> though a new token was obtained successfully. The failure appears at the 24h 
> HDFS token deadline as SecretManager$InvalidToken.
> h2. Observed failure
> This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
> ZooKeeper-based high availability and HDFS-backed checkpoint storage. The 
> primary failure was in the checkpoint storage path:
> {noformat}
> CheckpointCoordinator.initializeCheckpointLocation
>   -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
>   -> HadoopFileSystem.mkdirs
>   -> DFSClient.mkdirs
> {noformat}
> The checkpoint storage client reported InvalidToken for the startup token 
> after the refresh had created a new HDFS token and the receiver reported 
> success. The same JVM also showed a FileSystemJobResultStore stack using the 
> expired token during cleanup. The job-result store is a secondary long-lived 
> client with the same failure mode, not the primary trigger.
> Not every Kubernetes application-mode job is affected. The failure is 
> timing-dependent: it requires a long-lived Hadoop client to have captured one 
> Subject/UGI lifecycle while the refresh path later resolves to a different 
> one.
> Installation idempotency is out of scope. A diagnostic build that skipped the 
> second security installation still failed at the 24h HDFS token deadline.
> h2. Root cause
> HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() 
> of the thread executing the refresh. That current UGI is resolved from the 
> thread's effective AccessControlContext and is not guaranteed to be the UGI 
> held by HadoopSecurityContext or the UGI captured by a long-lived 
> checkpoint-storage or job-result-store DFSClient.
> The controlled Kubernetes probe observed this path:
> {noformat}
> provider thread                 = default-dispatcher-12
> freshUGI Subject                = 817147333
> login/security context Subject  = 1333938290
> current Subject before doAs    = 1333938290
> current Subject inside doAs    = 817147333
> current Subject after doAs     = 1333938290
> FJP workers created during the provider's blocked RPC:
>   default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12
> later cluster-io worker:
>   cluster-io-thread-3, creator=default-dispatcher-15, subjectId=817147333
> {noformat}
> The fresh Subject is not created by HadoopSecurityContext or by the 
> dispatcher. It belongs to the UGI returned by 
> {{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls 
> {{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}. 
> That call performs a Kerberos login and returns a newly constructed UGI for 
> the same principal; it does not replace the process login user.
> During {{freshUGI.doAs(...)}}, {{fs.addDelegationTokens(...)}} performed an 
> HDFS delegation-token RPC and blocked in {{CompletableFuture.get()}}. Because 
> the caller was a ForkJoinWorkerThread, ForkJoinPool entered its managed-block 
> compensation path and created {{default-dispatcher-13}}, 
> {{default-dispatcher-14}}, and {{default-dispatcher-15}} while the fresh 
> Subject was active. The Thread constructors copied the creator's effective 
> AccessControlContext, including the fresh Subject. After {{doAs}} returned, 
> {{getCurrentUser()}} on the creator resolved to the login Subject again, but 
> the newly created workers retained the inherited fresh Subject.
> Later, {{default-dispatcher-15}} created {{cluster-io-thread-3}}; the probe 
> recorded {{subjectId=817147333}} for that worker while {{getLoginUser()}} and 
> {{HadoopSecurityContext}} still referred to the login Subject {{1333938290}}. 
> A receiver task scheduled on such a worker can therefore add credentials to 
> the fresh Subject rather than the UGI captured by the long-lived client. This 
> is a per-thread inherited security context, not a global mutation of 
> {{UserGroupInformation}} or the process login user.
> The provider stack identifies this invocation as {{ResourceManager.onStart}} 
> -> {{DefaultDelegationTokenManager.start()}} -> {{startTokensUpdate()}}, i.e. 
> the initial token acquisition. The later scheduled refresh is submitted 
> through the cluster-io executor. The startup FJP expansion is observed; the 
> same mechanism is possible for any provider {{doAs}} on a 
> ForkJoinWorkerThread with a blocking HDFS RPC, but that later path has not 
> been directly reproduced in this probe.
> h2. YARN status and scope
> In a controlled YARN application-mode probe, the provider also ran on a Pekko 
> default-dispatcher worker. The provider window showed {{poolSize=3}} / 
> {{active=2}} before and after, with no {{tryCompensate}} or {{createWorker}}; 
> the current Subject after the provider returned was the login Subject, while 
> the provider's internal {{freshUGI}} Subject was a different value. YARN did 
> show ForkJoinPool expansion during the separate JobGraph-generation startup 
> path, so this is not evidence that YARN is immune.
> This is one YARN sample; the provider's {{CompletableFuture}} state was not 
> instrumented, so the YARN trigger is not established. YARN is not listed as a 
> confirmed reproduction, but it is not claimed to be exempt.
> h2. Proposed change
> This issue addresses the case where a token is obtained successfully but the 
> refreshed credentials do not reach long-lived Hadoop clients, primarily 
> checkpoint storage.
> * Add an accessor for the UGI held by HadoopSecurityContext.
> * In HadoopDelegationTokenReceiver, add refreshed credentials to the 
> security-context UGI, the current UGI, and the login UGI. Deduplicate the 
> target UGI references by object identity; this prevents duplicate 
> {{addCredentials}} calls for the same wrapper, but does not merge distinct 
> UGI wrappers that reference the same Subject. The security-context UGI is the 
> primary target; the other two preserve non-secured deployments and previous 
> behavior.
> * Add tests covering (a) a refresh thread whose current user differs from the 
> login user, and (b) a security-context UGI distinct from the current/login 
> user; verify that all applicable targets receive the credentials.
> Out of scope for this issue:
> * Preventing HDFS client initialization or blocking RPCs from occurring 
> inside {{freshUGI.doAs(...)}} or the provider's ForkJoinPool worker.
> * Closing or rebinding FileSystem instances created under a transient fresh 
> Subject; this is a separate client-lifecycle cleanup.
> * SecurityUtils.install idempotency.
> The proposed receiver change does not rely solely on the refresh thread's 
> inherited Subject. It does not by itself guarantee that a client already 
> created under a different Subject is rebound. That remains a separate 
> lifecycle concern.
> h2. Testing
> * A controlled Kubernetes application-mode probe recorded the chain from 
> {{fs.addDelegationTokens()}} through Hadoop ipc {{Client.getRpcResponse()}}, 
> {{CompletableFuture.get()}}, {{ForkJoinPool.managedBlock()}}, 
> {{tryCompensate()}}, and Pekko {{ThreadFactory.newThread()}}. The creator and 
> the new dispatcher workers had the same fresh Subject identity.
> * The same probe recorded {{cluster-io-thread-3}} created by 
> {{default-dispatcher-15}} with {{subjectId=817147333}}, while the login and 
> security-context Subjects remained {{1333938290}}.
> * Trigger multiple post-startup token refreshes in a long-running job and 
> verify which executor runs the provider, whether new ForkJoinPool workers are 
> created, and which Subject the receiver's current user resolves to.
> * The controlled YARN application-mode probe showed no {{tryCompensate}} or 
> {{createWorker}} in the provider window. The separate JobGraph-generation 
> startup path did create FJP workers, confirming that YARN can enter the same 
> FJP mechanism under different startup work; the provider's 
> {{CompletableFuture}} blocking state remains unverified.
> * A deployed application-mode job should survive the 24h token deadline and 
> continue checkpointing without SecretManager$InvalidToken. Verify that the 
> checkpoint-storage DFSClient receives the new token, and verify the 
> job-result-store client separately because it can capture a different UGI.
> h2. Acceptance criteria
> * A token refresh updates the UGI used by checkpoint storage and other 
> long-lived Hadoop clients.
> * Refresh delivers credentials to the security-context UGI regardless of 
> which Subject a pooled dispatcher or cluster-io worker inherited.
> * A long-running application-mode job survives the 24h HDFS token deadline.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to