[ 
https://issues.apache.org/jira/browse/FLINK-41435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

LuChengxiang updated FLINK-41435:
---------------------------------
    Description: 
h2. Summary

Flink periodically obtains HDFS delegation tokens and distributes them through 
HadoopDelegationTokenReceiver. The receiver currently adds new credentials to 
UserGroupInformation.getCurrentUser() on the thread that performs the refresh. 
Long-lived Hadoop clients, especially checkpoint storage, can capture a 
UGI/Subject when their DFSClient is created and use that captured identity for 
later RPC connections.

If the refresh path resolves to a different current UGI from the one captured 
by a long-lived client, the client can continue using its startup token even 
though a new token was obtained successfully. The failure appears at the 24h 
HDFS token deadline as SecretManager$InvalidToken.

The observed Kubernetes reproduction is in the initial token acquisition path. 
The receiver mismatch is not limited to that path: it applies whenever a 
refresh task resolves to a different current UGI. In this issue, "UGI" means 
the Hadoop UserGroupInformation wrapper, while "Subject" means the identity 
carried by that UGI; identity comparisons between wrappers use Subject identity.

h2. Observed failure

This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The 
primary failure was in the checkpoint storage path:

{noformat}
CheckpointCoordinator.initializeCheckpointLocation
  -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
  -> HadoopFileSystem.mkdirs
  -> DFSClient.mkdirs
{noformat}

The checkpoint storage client reported InvalidToken for the startup token after 
the refresh had created a new HDFS token and the receiver reported success. The 
same JVM also showed a FileSystemJobResultStore stack using the expired token 
during cleanup. The job-result store is a secondary long-lived client with the 
same failure mode, not the primary trigger.

Not every Kubernetes application-mode job is affected. The failure is 
timing-dependent: it requires a long-lived Hadoop client to have captured one 
Subject/UGI lifecycle while the refresh path later resolves to a different one.

Installation idempotency is out of scope. A diagnostic build that skipped the 
second security installation still failed at the 24h HDFS token deadline.

h2. Root cause

HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() of 
the thread executing the refresh. That current UGI is resolved from the 
thread's effective AccessControlContext and is not guaranteed to be the UGI 
held by HadoopSecurityContext or the UGI captured by a long-lived 
checkpoint-storage or job-result-store DFSClient.

The controlled Kubernetes probe observed the initial token acquisition path 
({{ResourceManager.onStart}} -> {{DefaultDelegationTokenManager.start()}} -> 
{{startTokensUpdate()}}), not a later scheduled refresh; the later scheduled 
refresh is submitted through the cluster-io executor. The startup FJP expansion 
is observed; the same mechanism is possible for any provider {{doAs}} on a 
ForkJoinWorkerThread with a blocking HDFS RPC, but that later path has not been 
directly reproduced in this probe.

{noformat}
provider thread                 = default-dispatcher-12
freshUGI Subject                = 817147333
login/HadoopSecurityContext Subject = 1333938290

current Subject before doAs    = 1333938290
current Subject inside doAs    = 817147333
current Subject after doAs     = 1333938290

FJP workers created during the provider's blocked RPC:
  default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12

later cluster-io worker:
  cluster-io-thread-3, creator=default-dispatcher-15
  current Subject                 = 817147333
  login/HadoopSecurityContext Subject = 1333938290
{noformat}

Here {{HadoopSecurityContext Subject}} means the Subject of the UGI held by the 
installed {{HadoopSecurityContext}}. In this probe it is the same Subject 
identity as the login user; that is not guaranteed in general.

The fresh Subject is not created by HadoopSecurityContext or by the dispatcher. 
It belongs to the UGI returned by 
{{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls 
{{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}. 
That call performs a Kerberos login and returns a newly constructed UGI for the 
same principal; it does not replace the process login user.

During {{freshUGI.doAs(...)}}, {{fs.addDelegationTokens(...)}} performed an 
HDFS delegation-token RPC and blocked in {{CompletableFuture.get()}}. Because 
the caller was a ForkJoinWorkerThread, ForkJoinPool entered its managed-block 
compensation path and created new dispatcher workers while the fresh Subject 
was active. The Thread constructors copied the creator's effective 
AccessControlContext, including the fresh Subject. After {{doAs}} returned, 
current on the creator resolved to the login Subject again, but the new workers 
retained the inherited fresh Subject.

Here {{fs}} refers to a FileSystem collected by 
{{HadoopFSDelegationTokenProvider.getFileSystemsToAccess()}}: the default 
Hadoop FileSystem from {{fs.defaultFS}}, plus any paths configured in 
{{security.kerberos.access.hadoopFileSystems}} (and the YARN staging directory 
when applicable). It is not necessarily the checkpoint-storage client object.

The dispatcher worker created during the {{doAs}} window later created 
{{cluster-io-thread-3}}. The probe recorded that this worker's current Subject 
was the fresh Subject while its login/HadoopSecurityContext Subject remained 
{{1333938290}}. Because the current receiver adds credentials to 
{{UserGroupInformation.getCurrentUser()}}, a receiver invocation on that worker 
would target the fresh Subject rather than the UGI captured by the long-lived 
client. This is a per-thread inherited security context, not a global mutation 
of {{UserGroupInformation}} or the process login user.

h2. YARN status and scope

In a controlled YARN application-mode probe, the provider also ran on a Pekko 
default-dispatcher worker. The provider's internal {{freshUGI}} Subject 
differed from the login Subject, and after {{doAs}} returned the creator's 
current Subject was the login Subject again. That lifecycle is also observed on 
Kubernetes, so it is not the platform difference. The distinguishing 
observation is that the YARN provider window stayed at {{poolSize=3}} / 
{{active=2}} with no {{tryCompensate}} or {{createWorker}}, so no new 
dispatcher worker was created while the fresh Subject was active. YARN did show 
ForkJoinPool expansion during the separate JobGraph-generation startup path, so 
this is not evidence that YARN is immune.

This is one YARN sample; the provider's {{CompletableFuture}} state was not 
instrumented, so the YARN trigger is not established. YARN is not listed as a 
confirmed reproduction, but it is not claimed to be exempt.

h2. Proposed change

This issue addresses the case where a token is obtained successfully but the 
refreshed credentials do not reach long-lived Hadoop clients, primarily 
checkpoint storage.

* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, add refreshed credentials to the UGI held 
by HadoopSecurityContext, the current UGI, and the login UGI. Deduplicate the 
target UGI references by object identity; this prevents duplicate 
{{addCredentials}} calls for the same wrapper, but does not merge distinct UGI 
wrappers that reference the same Subject. The UGI held by HadoopSecurityContext 
is the primary target; the other two preserve non-secured deployments and 
previous behavior.
* Add tests covering (a) a refresh thread whose current user differs from the 
login user, and (b) a UGI held by HadoopSecurityContext distinct from the 
current/login user; verify that all applicable targets receive the credentials.

Out of scope for this issue:

* Preventing HDFS client initialization or blocking RPCs from occurring inside 
{{freshUGI.doAs(...)}} or the provider's ForkJoinPool worker.
* Closing or rebinding FileSystem instances created under a transient fresh 
Subject; this is a separate client-lifecycle cleanup.
* SecurityUtils.install idempotency.

The proposed receiver change does not rely solely on the refresh thread's 
inherited Subject. It does not by itself guarantee that a client already 
created under a different Subject is rebound. That remains a separate lifecycle 
concern.

h2. Testing

* A controlled Kubernetes application-mode probe recorded the chain from 
{{fs.addDelegationTokens()}} through Hadoop ipc {{Client.getRpcResponse()}}, 
{{CompletableFuture.get()}}, {{ForkJoinPool.managedBlock()}}, 
{{tryCompensate()}}, and Pekko {{ThreadFactory.newThread()}}. The creator and 
the new dispatcher workers had the same fresh Subject identity.
* The same probe recorded {{cluster-io-thread-3}} created by 
{{default-dispatcher-15}} with {{subjectId=817147333}}, while the 
login/HadoopSecurityContext Subject remained {{1333938290}}.
* Trigger multiple post-startup token refreshes in a long-running job and 
verify which executor runs the provider, whether new ForkJoinPool workers are 
created, and which Subject the receiver's current user resolves to.
* Repeat the YARN provider-window probe while instrumenting the provider's 
{{CompletableFuture}} state; the current YARN sample is summarized in "YARN 
status and scope".
* A deployed application-mode job should survive the 24h token deadline and 
continue checkpointing without SecretManager$InvalidToken. Verify that the 
checkpoint-storage DFSClient receives the new token, and verify the 
job-result-store client separately because it can capture a different UGI.

h2. Acceptance criteria

* A token refresh updates the UGI used by checkpoint storage and other 
long-lived Hadoop clients.
* Refresh delivers credentials to the UGI held by HadoopSecurityContext 
regardless of which Subject a pooled dispatcher or cluster-io worker inherited.
* A long-running application-mode job survives the 24h HDFS token deadline.


  was:
h2. Summary

Flink periodically obtains HDFS delegation tokens and distributes them through 
HadoopDelegationTokenReceiver. The receiver currently adds new credentials to 
UserGroupInformation.getCurrentUser() on the thread that performs the refresh. 
Long-lived Hadoop clients, especially checkpoint storage, can capture a 
UGI/Subject when their DFSClient is created and use that captured identity for 
later RPC connections.

If the refresh path resolves to a different current UGI from the one captured 
by a long-lived client, the client can continue using its startup token even 
though a new token was obtained successfully. The failure appears at the 24h 
HDFS token deadline as SecretManager$InvalidToken.

The observed Kubernetes reproduction is in the initial token acquisition path. 
The receiver mismatch is not limited to that path: it applies whenever a 
refresh task resolves to a different current UGI. In this issue, "UGI" means 
the Hadoop UserGroupInformation wrapper, while "Subject" means the identity 
carried by that UGI; identity comparisons between wrappers use Subject identity.

h2. Observed failure

This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The 
primary failure was in the checkpoint storage path:

{noformat}
CheckpointCoordinator.initializeCheckpointLocation
  -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
  -> HadoopFileSystem.mkdirs
  -> DFSClient.mkdirs
{noformat}

The checkpoint storage client reported InvalidToken for the startup token after 
the refresh had created a new HDFS token and the receiver reported success. The 
same JVM also showed a FileSystemJobResultStore stack using the expired token 
during cleanup. The job-result store is a secondary long-lived client with the 
same failure mode, not the primary trigger.

Not every Kubernetes application-mode job is affected. The failure is 
timing-dependent: it requires a long-lived Hadoop client to have captured one 
Subject/UGI lifecycle while the refresh path later resolves to a different one.

Installation idempotency is out of scope. A diagnostic build that skipped the 
second security installation still failed at the 24h HDFS token deadline.

h2. Root cause

HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() of 
the thread executing the refresh. That current UGI is resolved from the 
thread's effective AccessControlContext and is not guaranteed to be the UGI 
held by HadoopSecurityContext or the UGI captured by a long-lived 
checkpoint-storage or job-result-store DFSClient.

The controlled Kubernetes probe observed the initial token acquisition path 
({{ResourceManager.onStart}} -> {{DefaultDelegationTokenManager.start()}} -> 
{{startTokensUpdate()}}), not a later scheduled refresh; the later scheduled 
refresh is submitted through the cluster-io executor. The startup FJP expansion 
is observed; the same mechanism is possible for any provider {{doAs}} on a 
ForkJoinWorkerThread with a blocking HDFS RPC, but that later path has not been 
directly reproduced in this probe.

{noformat}
provider thread                 = default-dispatcher-12
freshUGI Subject                = 817147333
login/HadoopSecurityContext Subject = 1333938290

current Subject before doAs    = 1333938290
current Subject inside doAs    = 817147333
current Subject after doAs     = 1333938290

FJP workers created during the provider's blocked RPC:
  default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12

later cluster-io worker:
  cluster-io-thread-3, creator=default-dispatcher-15
  current Subject                 = 817147333
  login/HadoopSecurityContext Subject = 1333938290
{noformat}

Here {{HadoopSecurityContext Subject}} means the Subject of the UGI held by the 
installed {{HadoopSecurityContext}}. In this probe it is the same Subject 
identity as the login user; that is not guaranteed in general.

The fresh Subject is not created by HadoopSecurityContext or by the dispatcher. 
It belongs to the UGI returned by 
{{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls 
{{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}. 
That call performs a Kerberos login and returns a newly constructed UGI for the 
same principal; it does not replace the process login user.

During {{freshUGI.doAs(...)}}, {{fs.addDelegationTokens(...)}} performed an 
HDFS delegation-token RPC and blocked in {{CompletableFuture.get()}}. Because 
the caller was a ForkJoinWorkerThread, ForkJoinPool entered its managed-block 
compensation path and created new dispatcher workers while the fresh Subject 
was active. The Thread constructors copied the creator's effective 
AccessControlContext, including the fresh Subject. After {{doAs}} returned, 
current on the creator resolved to the login Subject again, but the new workers 
retained the inherited fresh Subject.

Here {{fs}} refers to a FileSystem collected by 
{{HadoopFSDelegationTokenProvider.getFileSystemsToAccess()}}: the default 
Hadoop FileSystem from {{fs.defaultFS}}, plus any paths configured in 
{{security.kerberos.access.hadoopFileSystems}} (and the YARN staging directory 
when applicable). It is not necessarily the checkpoint-storage client object.

The dispatcher worker created during the {{doAs}} window later created 
{{cluster-io-thread-3}}. The probe recorded that this worker's current Subject 
was the fresh Subject while its login/HadoopSecurityContext Subject remained 
{{1333938290}}. Because the current receiver adds credentials to 
{{UserGroupInformation.getCurrentUser()}}, a receiver invocation on that worker 
would target the fresh Subject rather than the UGI captured by the long-lived 
client. This is a per-thread inherited security context, not a global mutation 
of {{UserGroupInformation}} or the process login user.

h2. YARN status and scope

In a controlled YARN application-mode probe, the provider also ran on a Pekko 
default-dispatcher worker. The provider window showed {{poolSize=3}} / 
{{active=2}} before and after, with no {{tryCompensate}} or {{createWorker}}; 
the current Subject after the provider returned was the login Subject, while 
the provider's internal {{freshUGI}} Subject was a different value. YARN did 
show ForkJoinPool expansion during the separate JobGraph-generation startup 
path, so this is not evidence that YARN is immune.

This is one YARN sample; the provider's {{CompletableFuture}} state was not 
instrumented, so the YARN trigger is not established. YARN is not listed as a 
confirmed reproduction, but it is not claimed to be exempt.

h2. Proposed change

This issue addresses the case where a token is obtained successfully but the 
refreshed credentials do not reach long-lived Hadoop clients, primarily 
checkpoint storage.

* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, add refreshed credentials to the UGI held 
by HadoopSecurityContext, the current UGI, and the login UGI. Deduplicate the 
target UGI references by object identity; this prevents duplicate 
{{addCredentials}} calls for the same wrapper, but does not merge distinct UGI 
wrappers that reference the same Subject. The UGI held by HadoopSecurityContext 
is the primary target; the other two preserve non-secured deployments and 
previous behavior.
* Add tests covering (a) a refresh thread whose current user differs from the 
login user, and (b) a UGI held by HadoopSecurityContext distinct from the 
current/login user; verify that all applicable targets receive the credentials.

Out of scope for this issue:

* Preventing HDFS client initialization or blocking RPCs from occurring inside 
{{freshUGI.doAs(...)}} or the provider's ForkJoinPool worker.
* Closing or rebinding FileSystem instances created under a transient fresh 
Subject; this is a separate client-lifecycle cleanup.
* SecurityUtils.install idempotency.

The proposed receiver change does not rely solely on the refresh thread's 
inherited Subject. It does not by itself guarantee that a client already 
created under a different Subject is rebound. That remains a separate lifecycle 
concern.

h2. Testing

* A controlled Kubernetes application-mode probe recorded the chain from 
{{fs.addDelegationTokens()}} through Hadoop ipc {{Client.getRpcResponse()}}, 
{{CompletableFuture.get()}}, {{ForkJoinPool.managedBlock()}}, 
{{tryCompensate()}}, and Pekko {{ThreadFactory.newThread()}}. The creator and 
the new dispatcher workers had the same fresh Subject identity.
* The same probe recorded {{cluster-io-thread-3}} created by 
{{default-dispatcher-15}} with {{subjectId=817147333}}, while the 
login/HadoopSecurityContext Subject remained {{1333938290}}.
* Trigger multiple post-startup token refreshes in a long-running job and 
verify which executor runs the provider, whether new ForkJoinPool workers are 
created, and which Subject the receiver's current user resolves to.
* Repeat the YARN provider-window probe while instrumenting the provider's 
{{CompletableFuture}} state; the current YARN sample is summarized in "YARN 
status and scope".
* A deployed application-mode job should survive the 24h token deadline and 
continue checkpointing without SecretManager$InvalidToken. Verify that the 
checkpoint-storage DFSClient receives the new token, and verify the 
job-result-store client separately because it can capture a different UGI.

h2. Acceptance criteria

* A token refresh updates the UGI used by checkpoint storage and other 
long-lived Hadoop clients.
* Refresh delivers credentials to the UGI held by HadoopSecurityContext 
regardless of which Subject a pooled dispatcher or cluster-io worker inherited.
* A long-running application-mode job survives the 24h HDFS token deadline.



> Hadoop delegation token refresh can target a different UGI than Hadoop clients
> ------------------------------------------------------------------------------
>
>                 Key: FLINK-41435
>                 URL: https://issues.apache.org/jira/browse/FLINK-41435
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Configuration
>    Affects Versions: 2.0.0, 1.19.0, 1.20.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0
>         Environment: Flink 1.19.0 and later (1.19.x, 1.20.x, 2.0 through 
> 2.4), plus master (2.5-SNAPSHOT); Kubernetes and standalone application mode 
> with Kerberos and HDFS delegation tokens. The mismatch is more likely when 
> FileSystemJobResultStore and the token manager share the cluster-io executor 
> (FLINK-27204).
>            Reporter: LuChengxiang
>            Priority: Major
>
> h2. Summary
> Flink periodically obtains HDFS delegation tokens and distributes them 
> through HadoopDelegationTokenReceiver. The receiver currently adds new 
> credentials to UserGroupInformation.getCurrentUser() on the thread that 
> performs the refresh. Long-lived Hadoop clients, especially checkpoint 
> storage, can capture a UGI/Subject when their DFSClient is created and use 
> that captured identity for later RPC connections.
> If the refresh path resolves to a different current UGI from the one captured 
> by a long-lived client, the client can continue using its startup token even 
> though a new token was obtained successfully. The failure appears at the 24h 
> HDFS token deadline as SecretManager$InvalidToken.
> The observed Kubernetes reproduction is in the initial token acquisition 
> path. The receiver mismatch is not limited to that path: it applies whenever 
> a refresh task resolves to a different current UGI. In this issue, "UGI" 
> means the Hadoop UserGroupInformation wrapper, while "Subject" means the 
> identity carried by that UGI; identity comparisons between wrappers use 
> Subject identity.
> h2. Observed failure
> This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
> ZooKeeper-based high availability and HDFS-backed checkpoint storage. The 
> primary failure was in the checkpoint storage path:
> {noformat}
> CheckpointCoordinator.initializeCheckpointLocation
>   -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
>   -> HadoopFileSystem.mkdirs
>   -> DFSClient.mkdirs
> {noformat}
> The checkpoint storage client reported InvalidToken for the startup token 
> after the refresh had created a new HDFS token and the receiver reported 
> success. The same JVM also showed a FileSystemJobResultStore stack using the 
> expired token during cleanup. The job-result store is a secondary long-lived 
> client with the same failure mode, not the primary trigger.
> Not every Kubernetes application-mode job is affected. The failure is 
> timing-dependent: it requires a long-lived Hadoop client to have captured one 
> Subject/UGI lifecycle while the refresh path later resolves to a different 
> one.
> Installation idempotency is out of scope. A diagnostic build that skipped the 
> second security installation still failed at the 24h HDFS token deadline.
> h2. Root cause
> HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() 
> of the thread executing the refresh. That current UGI is resolved from the 
> thread's effective AccessControlContext and is not guaranteed to be the UGI 
> held by HadoopSecurityContext or the UGI captured by a long-lived 
> checkpoint-storage or job-result-store DFSClient.
> The controlled Kubernetes probe observed the initial token acquisition path 
> ({{ResourceManager.onStart}} -> {{DefaultDelegationTokenManager.start()}} -> 
> {{startTokensUpdate()}}), not a later scheduled refresh; the later scheduled 
> refresh is submitted through the cluster-io executor. The startup FJP 
> expansion is observed; the same mechanism is possible for any provider 
> {{doAs}} on a ForkJoinWorkerThread with a blocking HDFS RPC, but that later 
> path has not been directly reproduced in this probe.
> {noformat}
> provider thread                 = default-dispatcher-12
> freshUGI Subject                = 817147333
> login/HadoopSecurityContext Subject = 1333938290
> current Subject before doAs    = 1333938290
> current Subject inside doAs    = 817147333
> current Subject after doAs     = 1333938290
> FJP workers created during the provider's blocked RPC:
>   default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12
> later cluster-io worker:
>   cluster-io-thread-3, creator=default-dispatcher-15
>   current Subject                 = 817147333
>   login/HadoopSecurityContext Subject = 1333938290
> {noformat}
> Here {{HadoopSecurityContext Subject}} means the Subject of the UGI held by 
> the installed {{HadoopSecurityContext}}. In this probe it is the same Subject 
> identity as the login user; that is not guaranteed in general.
> The fresh Subject is not created by HadoopSecurityContext or by the 
> dispatcher. It belongs to the UGI returned by 
> {{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls 
> {{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}. 
> That call performs a Kerberos login and returns a newly constructed UGI for 
> the same principal; it does not replace the process login user.
> During {{freshUGI.doAs(...)}}, {{fs.addDelegationTokens(...)}} performed an 
> HDFS delegation-token RPC and blocked in {{CompletableFuture.get()}}. Because 
> the caller was a ForkJoinWorkerThread, ForkJoinPool entered its managed-block 
> compensation path and created new dispatcher workers while the fresh Subject 
> was active. The Thread constructors copied the creator's effective 
> AccessControlContext, including the fresh Subject. After {{doAs}} returned, 
> current on the creator resolved to the login Subject again, but the new 
> workers retained the inherited fresh Subject.
> Here {{fs}} refers to a FileSystem collected by 
> {{HadoopFSDelegationTokenProvider.getFileSystemsToAccess()}}: the default 
> Hadoop FileSystem from {{fs.defaultFS}}, plus any paths configured in 
> {{security.kerberos.access.hadoopFileSystems}} (and the YARN staging 
> directory when applicable). It is not necessarily the checkpoint-storage 
> client object.
> The dispatcher worker created during the {{doAs}} window later created 
> {{cluster-io-thread-3}}. The probe recorded that this worker's current 
> Subject was the fresh Subject while its login/HadoopSecurityContext Subject 
> remained {{1333938290}}. Because the current receiver adds credentials to 
> {{UserGroupInformation.getCurrentUser()}}, a receiver invocation on that 
> worker would target the fresh Subject rather than the UGI captured by the 
> long-lived client. This is a per-thread inherited security context, not a 
> global mutation of {{UserGroupInformation}} or the process login user.
> h2. YARN status and scope
> In a controlled YARN application-mode probe, the provider also ran on a Pekko 
> default-dispatcher worker. The provider's internal {{freshUGI}} Subject 
> differed from the login Subject, and after {{doAs}} returned the creator's 
> current Subject was the login Subject again. That lifecycle is also observed 
> on Kubernetes, so it is not the platform difference. The distinguishing 
> observation is that the YARN provider window stayed at {{poolSize=3}} / 
> {{active=2}} with no {{tryCompensate}} or {{createWorker}}, so no new 
> dispatcher worker was created while the fresh Subject was active. YARN did 
> show ForkJoinPool expansion during the separate JobGraph-generation startup 
> path, so this is not evidence that YARN is immune.
> This is one YARN sample; the provider's {{CompletableFuture}} state was not 
> instrumented, so the YARN trigger is not established. YARN is not listed as a 
> confirmed reproduction, but it is not claimed to be exempt.
> h2. Proposed change
> This issue addresses the case where a token is obtained successfully but the 
> refreshed credentials do not reach long-lived Hadoop clients, primarily 
> checkpoint storage.
> * Add an accessor for the UGI held by HadoopSecurityContext.
> * In HadoopDelegationTokenReceiver, add refreshed credentials to the UGI held 
> by HadoopSecurityContext, the current UGI, and the login UGI. Deduplicate the 
> target UGI references by object identity; this prevents duplicate 
> {{addCredentials}} calls for the same wrapper, but does not merge distinct 
> UGI wrappers that reference the same Subject. The UGI held by 
> HadoopSecurityContext is the primary target; the other two preserve 
> non-secured deployments and previous behavior.
> * Add tests covering (a) a refresh thread whose current user differs from the 
> login user, and (b) a UGI held by HadoopSecurityContext distinct from the 
> current/login user; verify that all applicable targets receive the 
> credentials.
> Out of scope for this issue:
> * Preventing HDFS client initialization or blocking RPCs from occurring 
> inside {{freshUGI.doAs(...)}} or the provider's ForkJoinPool worker.
> * Closing or rebinding FileSystem instances created under a transient fresh 
> Subject; this is a separate client-lifecycle cleanup.
> * SecurityUtils.install idempotency.
> The proposed receiver change does not rely solely on the refresh thread's 
> inherited Subject. It does not by itself guarantee that a client already 
> created under a different Subject is rebound. That remains a separate 
> lifecycle concern.
> h2. Testing
> * A controlled Kubernetes application-mode probe recorded the chain from 
> {{fs.addDelegationTokens()}} through Hadoop ipc {{Client.getRpcResponse()}}, 
> {{CompletableFuture.get()}}, {{ForkJoinPool.managedBlock()}}, 
> {{tryCompensate()}}, and Pekko {{ThreadFactory.newThread()}}. The creator and 
> the new dispatcher workers had the same fresh Subject identity.
> * The same probe recorded {{cluster-io-thread-3}} created by 
> {{default-dispatcher-15}} with {{subjectId=817147333}}, while the 
> login/HadoopSecurityContext Subject remained {{1333938290}}.
> * Trigger multiple post-startup token refreshes in a long-running job and 
> verify which executor runs the provider, whether new ForkJoinPool workers are 
> created, and which Subject the receiver's current user resolves to.
> * Repeat the YARN provider-window probe while instrumenting the provider's 
> {{CompletableFuture}} state; the current YARN sample is summarized in "YARN 
> status and scope".
> * A deployed application-mode job should survive the 24h token deadline and 
> continue checkpointing without SecretManager$InvalidToken. Verify that the 
> checkpoint-storage DFSClient receives the new token, and verify the 
> job-result-store client separately because it can capture a different UGI.
> h2. Acceptance criteria
> * A token refresh updates the UGI used by checkpoint storage and other 
> long-lived Hadoop clients.
> * Refresh delivers credentials to the UGI held by HadoopSecurityContext 
> regardless of which Subject a pooled dispatcher or cluster-io worker 
> inherited.
> * A long-running application-mode job survives the 24h HDFS token deadline.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to