[
https://issues.apache.org/jira/browse/FLINK-41435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41435:
-----------------------------------
Labels: pull-request-available (was: )
> Hadoop delegation token refresh can target a different UGI than Hadoop clients
> ------------------------------------------------------------------------------
>
> Key: FLINK-41435
> URL: https://issues.apache.org/jira/browse/FLINK-41435
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
> Affects Versions: 2.0.0, 1.19.0, 1.20.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0
> Environment: Flink 1.19.0 and later (1.19.x, 1.20.x, 2.0 through
> 2.4), plus master (2.5-SNAPSHOT); Kubernetes and standalone application mode
> with Kerberos and HDFS delegation tokens. The mismatch is more likely when
> FileSystemJobResultStore and the token manager share the cluster-io executor
> (FLINK-27204).
> Reporter: LuChengxiang
> Priority: Major
> Labels: pull-request-available
>
> h2. Summary
> Flink periodically obtains HDFS delegation tokens and distributes them
> through HadoopDelegationTokenReceiver. The receiver adds refreshed
> credentials to UserGroupInformation.getCurrentUser() of the thread running
> the refresh. A long-lived Hadoop client, especially a checkpoint-storage
> DFSClient, may instead use the UGI/Subject captured when that client was
> created. If those identities differ, the refresh succeeds and the receiver
> reports success, but the new credentials are not delivered to the client's
> UGI. The client keeps using its startup token and fails at the 24h HDFS token
> deadline with SecretManager$InvalidToken.
> The defect is receiver target selection, not token acquisition or client
> binding. A pooled thread's current UGI can be inherited from a different
> Subject lifecycle than the UGI held by HadoopSecurityContext or the login
> user. The proposed fix targets the UGI held by HadoopSecurityContext, falling
> back to the login UGI, instead of a pooled thread's current UGI.
> Here "UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means
> the identity it carries; identity comparisons use Subject identity.
> h2. Observed failure
> This was observed in a Flink 1.20.5 Kubernetes application-mode job using
> ZooKeeper-based high availability and HDFS-backed checkpoint storage. The job
> used a Hadoop IPC client with the 3.5.0 CompletableFuture-based wait path.
> The primary failure was in the checkpoint storage path:
> {noformat}
> CheckpointCoordinator.initializeCheckpointLocation
> -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
> -> HadoopFileSystem.mkdirs
> -> DFSClient.mkdirs
> {noformat}
> After the 24h deadline, the checkpoint-storage client reported InvalidToken
> for its startup token even though refresh had obtained a new HDFS token and
> the receiver reported success.
> Not every application-mode job is affected. The failure is timing-dependent:
> a long-lived Hadoop client must capture one Subject/UGI lifecycle, and a
> later refresh receiver must run under a different one. Whether the provider's
> blocking wait enters FJP managed blocking determines whether a replacement
> worker can be created during the freshUGI.doAs(...) window.
> h2. Root cause
> HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser()
> of the refresh thread. getCurrentUser() resolves from the thread's effective
> AccessControlContext and is not guaranteed to be the UGI held by
> HadoopSecurityContext or the UGI captured by a long-lived checkpoint-storage
> DFSClient.
> In the observed failure, the checkpoint client's captured UGI had the
> login/HadoopSecurityContext Subject, while the receiver could run under a
> pooled thread's inherited Subject.
> h3. How the refresh thread's current UGI can differ
> The provider obtains tokens inside freshUGI.doAs(...). With Apache Hadoop
> 3.5.0 and later, Client.getRpcResponse waits on CompletableFuture.get()
> (HADOOP-19235). On a ForkJoinWorkerThread, CompletableFuture.get() enters
> ForkJoinPool.managedBlock(); if no idle worker is available and the pool has
> capacity, ForkJoinPool.tryCompensate() creates a replacement worker while
> freshUGI.doAs(...) is still effective.
> The Thread constructor copies the effective AccessControlContext into the new
> thread's inheritedAccessControlContext, so a worker created during that
> window retains the fresh Subject after doAs returns. A later cluster-io
> worker created by one of those workers inherits the same Subject. A receiver
> running on that cluster-io worker then resolves getCurrentUser() to the fresh
> Subject and writes credentials there, while the login and
> HadoopSecurityContext UGIs remain unchanged.
> {noformat}
> freshUGI.doAs(...)
> -> fs.addDelegationTokens(...)
> -> HDFS RPC in CompletableFuture.get() [Hadoop 3.5.0+]
> -> ForkJoinPool.managedBlock()
> -> tryCompensate()
> -> createWorker()
> -> new dispatcher worker inherits the fresh Subject
> -> later creates a cluster-io worker
> -> receiver runs with the fresh Subject
> {noformat}
> The lookup chain is:
> {noformat}
> Thread.inheritedAccessControlContext
> -> AccessController.getContext()
> -> Subject.getSubject(acc)
> -> UserGroupInformation.getCurrentUser()
> {noformat}
> getCurrentUser() reads the Subject from the current thread's ACC (the
> inherited ACC carries the SubjectDomainCombiner created by doAs), not from a
> global current user.
> In Apache Hadoop 3.3.x and 3.4.x, the same call waits in AsyncGet.Util.wait
> (Object.wait) instead. Object.wait is not an FJP managed block, so the
> provider wait itself does not call tryCompensate; the pool may still create
> workers for unrelated task submission or other managed blocking. This version
> boundary explains why the failure is not reproducible with those clients, but
> it does not make the receiver's getCurrentUser() assumption correct.
> h2. Proposed change
> This issue fixes token delivery for the observed long-lived-client case,
> primarily checkpoint storage.
> * Add an accessor for the UGI held by HadoopSecurityContext.
> * In HadoopDelegationTokenReceiver, select a single target UGI: the UGI held
> by HadoopSecurityContext when present; otherwise fall back to
> UserGroupInformation.getLoginUser(). Add refreshed credentials only to that
> UGI. Do not target UserGroupInformation.getCurrentUser().
> * Add unit tests for the receiver's context-first and login-fallback target
> selection.
> The change makes the delivery target deterministic. It does not rebind a
> client already created under a different Subject.
> h2. Scope and boundaries
> h3. Out of scope
> * Preventing HDFS client initialization or blocking RPCs from occurring
> inside freshUGI.doAs(...) or the provider's ForkJoinPool thread.
> * Closing or rebinding FileSystem instances created under a transient
> Subject; this is a separate client-lifecycle cleanup.
> * Changing the provider's getCurrentUser() usage. The provider already
> acquires tokens inside KerberosLoginProvider.doLoginAndReturnUGI(); its
> remaining getCurrentUser() use is a token-required pre-check and renewer-name
> lookup. A provider change is a separate follow-up if provider-side behavior
> is observed.
> * Double security installation and SecurityUtils.install idempotency. A
> diagnostic build where the second installation was skipped still failed at
> the 24h deadline.
> h3. Version context
> Flink: before 1.17, the legacy HadoopModule path added credentials directly
> to the login user; the HadoopDelegationTokenReceiver / getCurrentUser() path
> starts with 1.17. 1.18, 1.19, and 1.20 share that receiver behavior.
> Hadoop: Client.getRpcResponse uses CompletableFuture.get() starting with
> 3.5.0 (HADOOP-19235). 3.3.x and 3.4.x use AsyncGet.Util.wait. The trigger in
> this issue requires the CompletableFuture-based wait path; the receiver
> defect is independent of the Hadoop version.
> h3. Deployment scope
> The failure was observed in a Kubernetes application-mode job using the 3.5.0
> IPC behavior. It is not a Kubernetes-specific defect: the trigger is the
> client's wait path and the receiver's executing Subject, not the deployment
> type. Environments using the pre-3.5.0 AsyncGet.Util.wait path do not create
> FJP workers from the provider wait itself.
> h2. Testing
> * Add receiver unit tests for (a) HadoopSecurityContext UGI precedence, (b)
> login fallback when no HadoopSecurityContext is installed, and (c) the
> inherited current UGI not being targeted.
> * A deployed application-mode job using the Hadoop 3.5.0 IPC behavior should
> survive the 24h token deadline and continue checkpointing without
> SecretManager$InvalidToken. Verify that the checkpoint-storage DFSClient
> receives the new token.
> * Run a controlled application-mode probe that verifies the receiver updates
> the UGI selected by the new logic rather than a pooled thread's inherited
> current UGI; cover the TaskManager receiver path separately.
> h2. Acceptance criteria
> * A token refresh updates the UGI held by HadoopSecurityContext when present,
> otherwise the login UGI, and does not use a pooled thread's inherited current
> UGI; a checkpoint-storage client bound to the selected UGI receives the new
> token.
> * A long-running application-mode job using the Hadoop 3.5.0 IPC behavior
> survives the 24h HDFS token deadline.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)