[ 
https://issues.apache.org/jira/browse/FLINK-41435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-41435:
-----------------------------------
    Labels: pull-request-available  (was: )

> Hadoop delegation token refresh can target a different UGI than Hadoop clients
> ------------------------------------------------------------------------------
>
>                 Key: FLINK-41435
>                 URL: https://issues.apache.org/jira/browse/FLINK-41435
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Configuration
>    Affects Versions: 2.0.0, 1.19.0, 1.20.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0
>         Environment: Flink 1.19.0 and later (1.19.x, 1.20.x, 2.0 through 
> 2.4), plus master (2.5-SNAPSHOT); Kubernetes and standalone application mode 
> with Kerberos and HDFS delegation tokens. The mismatch is more likely when 
> FileSystemJobResultStore and the token manager share the cluster-io executor 
> (FLINK-27204).
>            Reporter: LuChengxiang
>            Priority: Major
>              Labels: pull-request-available
>
> h2. Summary
> Flink periodically obtains HDFS delegation tokens and distributes them 
> through HadoopDelegationTokenReceiver. The receiver adds refreshed 
> credentials to UserGroupInformation.getCurrentUser() of the thread running 
> the refresh. A long-lived Hadoop client, especially a checkpoint-storage 
> DFSClient, may instead use the UGI/Subject captured when that client was 
> created. If those identities differ, the refresh succeeds and the receiver 
> reports success, but the new credentials are not delivered to the client's 
> UGI. The client keeps using its startup token and fails at the 24h HDFS token 
> deadline with SecretManager$InvalidToken.
> The defect is receiver target selection, not token acquisition or client 
> binding. A pooled thread's current UGI can be inherited from a different 
> Subject lifecycle than the UGI held by HadoopSecurityContext or the login 
> user. The proposed fix targets the UGI held by HadoopSecurityContext, falling 
> back to the login UGI, instead of a pooled thread's current UGI.
> Here "UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means 
> the identity it carries; identity comparisons use Subject identity.
> h2. Observed failure
> This was observed in a Flink 1.20.5 Kubernetes application-mode job using 
> ZooKeeper-based high availability and HDFS-backed checkpoint storage. The job 
> used a Hadoop IPC client with the 3.5.0 CompletableFuture-based wait path. 
> The primary failure was in the checkpoint storage path:
> {noformat}
> CheckpointCoordinator.initializeCheckpointLocation
>   -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
>   -> HadoopFileSystem.mkdirs
>   -> DFSClient.mkdirs
> {noformat}
> After the 24h deadline, the checkpoint-storage client reported InvalidToken 
> for its startup token even though refresh had obtained a new HDFS token and 
> the receiver reported success.
> Not every application-mode job is affected. The failure is timing-dependent: 
> a long-lived Hadoop client must capture one Subject/UGI lifecycle, and a 
> later refresh receiver must run under a different one. Whether the provider's 
> blocking wait enters FJP managed blocking determines whether a replacement 
> worker can be created during the freshUGI.doAs(...) window.
> h2. Root cause
> HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() 
> of the refresh thread. getCurrentUser() resolves from the thread's effective 
> AccessControlContext and is not guaranteed to be the UGI held by 
> HadoopSecurityContext or the UGI captured by a long-lived checkpoint-storage 
> DFSClient.
> In the observed failure, the checkpoint client's captured UGI had the 
> login/HadoopSecurityContext Subject, while the receiver could run under a 
> pooled thread's inherited Subject.
> h3. How the refresh thread's current UGI can differ
> The provider obtains tokens inside freshUGI.doAs(...). With Apache Hadoop 
> 3.5.0 and later, Client.getRpcResponse waits on CompletableFuture.get() 
> (HADOOP-19235). On a ForkJoinWorkerThread, CompletableFuture.get() enters 
> ForkJoinPool.managedBlock(); if no idle worker is available and the pool has 
> capacity, ForkJoinPool.tryCompensate() creates a replacement worker while 
> freshUGI.doAs(...) is still effective.
> The Thread constructor copies the effective AccessControlContext into the new 
> thread's inheritedAccessControlContext, so a worker created during that 
> window retains the fresh Subject after doAs returns. A later cluster-io 
> worker created by one of those workers inherits the same Subject. A receiver 
> running on that cluster-io worker then resolves getCurrentUser() to the fresh 
> Subject and writes credentials there, while the login and 
> HadoopSecurityContext UGIs remain unchanged.
> {noformat}
> freshUGI.doAs(...)
>   -> fs.addDelegationTokens(...)
>   -> HDFS RPC in CompletableFuture.get()   [Hadoop 3.5.0+]
>   -> ForkJoinPool.managedBlock()
>   -> tryCompensate()
>   -> createWorker()
>   -> new dispatcher worker inherits the fresh Subject
>   -> later creates a cluster-io worker
>   -> receiver runs with the fresh Subject
> {noformat}
> The lookup chain is:
> {noformat}
> Thread.inheritedAccessControlContext
>   -> AccessController.getContext()
>   -> Subject.getSubject(acc)
>   -> UserGroupInformation.getCurrentUser()
> {noformat}
> getCurrentUser() reads the Subject from the current thread's ACC (the 
> inherited ACC carries the SubjectDomainCombiner created by doAs), not from a 
> global current user.
> In Apache Hadoop 3.3.x and 3.4.x, the same call waits in AsyncGet.Util.wait 
> (Object.wait) instead. Object.wait is not an FJP managed block, so the 
> provider wait itself does not call tryCompensate; the pool may still create 
> workers for unrelated task submission or other managed blocking. This version 
> boundary explains why the failure is not reproducible with those clients, but 
> it does not make the receiver's getCurrentUser() assumption correct.
> h2. Proposed change
> This issue fixes token delivery for the observed long-lived-client case, 
> primarily checkpoint storage.
> * Add an accessor for the UGI held by HadoopSecurityContext.
> * In HadoopDelegationTokenReceiver, select a single target UGI: the UGI held 
> by HadoopSecurityContext when present; otherwise fall back to 
> UserGroupInformation.getLoginUser(). Add refreshed credentials only to that 
> UGI. Do not target UserGroupInformation.getCurrentUser().
> * Add unit tests for the receiver's context-first and login-fallback target 
> selection.
> The change makes the delivery target deterministic. It does not rebind a 
> client already created under a different Subject.
> h2. Scope and boundaries
> h3. Out of scope
> * Preventing HDFS client initialization or blocking RPCs from occurring 
> inside freshUGI.doAs(...) or the provider's ForkJoinPool thread.
> * Closing or rebinding FileSystem instances created under a transient 
> Subject; this is a separate client-lifecycle cleanup.
> * Changing the provider's getCurrentUser() usage. The provider already 
> acquires tokens inside KerberosLoginProvider.doLoginAndReturnUGI(); its 
> remaining getCurrentUser() use is a token-required pre-check and renewer-name 
> lookup. A provider change is a separate follow-up if provider-side behavior 
> is observed.
> * Double security installation and SecurityUtils.install idempotency. A 
> diagnostic build where the second installation was skipped still failed at 
> the 24h deadline.
> h3. Version context
> Flink: before 1.17, the legacy HadoopModule path added credentials directly 
> to the login user; the HadoopDelegationTokenReceiver / getCurrentUser() path 
> starts with 1.17. 1.18, 1.19, and 1.20 share that receiver behavior.
> Hadoop: Client.getRpcResponse uses CompletableFuture.get() starting with 
> 3.5.0 (HADOOP-19235). 3.3.x and 3.4.x use AsyncGet.Util.wait. The trigger in 
> this issue requires the CompletableFuture-based wait path; the receiver 
> defect is independent of the Hadoop version.
> h3. Deployment scope
> The failure was observed in a Kubernetes application-mode job using the 3.5.0 
> IPC behavior. It is not a Kubernetes-specific defect: the trigger is the 
> client's wait path and the receiver's executing Subject, not the deployment 
> type. Environments using the pre-3.5.0 AsyncGet.Util.wait path do not create 
> FJP workers from the provider wait itself.
> h2. Testing
> * Add receiver unit tests for (a) HadoopSecurityContext UGI precedence, (b) 
> login fallback when no HadoopSecurityContext is installed, and (c) the 
> inherited current UGI not being targeted.
> * A deployed application-mode job using the Hadoop 3.5.0 IPC behavior should 
> survive the 24h token deadline and continue checkpointing without 
> SecretManager$InvalidToken. Verify that the checkpoint-storage DFSClient 
> receives the new token.
> * Run a controlled application-mode probe that verifies the receiver updates 
> the UGI selected by the new logic rather than a pooled thread's inherited 
> current UGI; cover the TaskManager receiver path separately.
> h2. Acceptance criteria
> * A token refresh updates the UGI held by HadoopSecurityContext when present, 
> otherwise the login UGI, and does not use a pooled thread's inherited current 
> UGI; a checkpoint-storage client bound to the selected UGI receives the new 
> token.
> * A long-running application-mode job using the Hadoop 3.5.0 IPC behavior 
> survives the 24h HDFS token deadline.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to