[
https://issues.apache.org/jira/browse/FLINK-41435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LuChengxiang updated FLINK-41435:
---------------------------------
Description:
h2. Summary
Flink periodically obtains HDFS delegation tokens and distributes them through
HadoopDelegationTokenReceiver. The receiver adds refreshed credentials to
UserGroupInformation.getCurrentUser() of the thread running the refresh. A
long-lived Hadoop client, especially a checkpoint-storage DFSClient, may
instead use the UGI/Subject captured when that client was created. If those
identities differ, the refresh succeeds and the receiver reports success, but
the new credentials are not delivered to the client's UGI. The client keeps
using its startup token and fails at the 24h HDFS token deadline with
SecretManager$InvalidToken.
The defect is receiver target selection, not token acquisition or client
binding. A pooled thread's current UGI can be inherited from a different
Subject lifecycle than the UGI held by HadoopSecurityContext or the login user.
The proposed fix targets the UGI held by HadoopSecurityContext, falling back to
the login UGI, instead of a pooled thread's current UGI.
Here "UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means
the identity it carries; identity comparisons use Subject identity.
h2. Observed failure
This was observed in a Flink 1.20.5 Kubernetes application-mode job using
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The job
used a Hadoop IPC client with the 3.5.0 CompletableFuture-based wait path. The
primary failure was in the checkpoint storage path:
{noformat}
CheckpointCoordinator.initializeCheckpointLocation
-> FsCheckpointStorageAccess.initializeLocationForCheckpoint
-> HadoopFileSystem.mkdirs
-> DFSClient.mkdirs
{noformat}
After the 24h deadline, the checkpoint-storage client reported InvalidToken for
its startup token even though refresh had obtained a new HDFS token and the
receiver reported success.
Not every application-mode job is affected. The failure is timing-dependent: a
long-lived Hadoop client must capture one Subject/UGI lifecycle, and a later
refresh receiver must run under a different one. The IPC wait path determines
whether the refresh thread can create such a divergent worker.
h2. Root cause
HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() of
the refresh thread. getCurrentUser() resolves from the thread's effective
AccessControlContext and is not guaranteed to be the UGI held by
HadoopSecurityContext or the UGI captured by a long-lived checkpoint-storage
DFSClient.
In the observed failure, the checkpoint client's captured UGI had the
login/HadoopSecurityContext Subject, while the receiver could run under a
pooled thread's inherited Subject.
h3. How the refresh thread's current UGI can differ
The provider obtains tokens inside freshUGI.doAs(...). With Apache Hadoop 3.5.0
and later, Client.getRpcResponse waits on CompletableFuture.get()
(HADOOP-19235). On a ForkJoinWorkerThread, CompletableFuture.get() enters
ForkJoinPool.managedBlock(); if no idle worker is available and the pool has
capacity, ForkJoinPool.tryCompensate() creates a replacement worker while
freshUGI.doAs(...) is still effective.
The Thread constructor copies the effective AccessControlContext into the new
thread's inheritedAccessControlContext, so a worker created during that window
retains the fresh Subject after doAs returns. A later cluster-io worker created
by one of those workers inherits the same Subject. A receiver running on that
cluster-io worker then resolves getCurrentUser() to the fresh Subject and
writes credentials there, while the login and HadoopSecurityContext UGIs remain
unchanged.
{noformat}
freshUGI.doAs(...)
-> fs.addDelegationTokens(...)
-> HDFS RPC in CompletableFuture.get() [Hadoop 3.5.0+]
-> ForkJoinPool.managedBlock()
-> tryCompensate()
-> createWorker()
-> new dispatcher worker inherits the fresh Subject
-> later creates a cluster-io worker
-> receiver runs with the fresh Subject
{noformat}
In Apache Hadoop 3.3.x and 3.4.x, the same call waits in AsyncGet.Util.wait
(Object.wait) instead. Object.wait is not an FJP managed block, so the pool
does not create a replacement worker; the provider thread blocks without
propagating the fresh Subject to a new thread. This version boundary explains
why the failure is not reproducible with those clients, but it does not make
the receiver's getCurrentUser() assumption correct.
h2. Proposed change
This issue fixes token delivery for the observed long-lived-client case,
primarily checkpoint storage.
* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, select a single target UGI: the UGI held by
HadoopSecurityContext when present; otherwise fall back to
UserGroupInformation.getLoginUser(). Add refreshed credentials only to that
UGI. Do not target UserGroupInformation.getCurrentUser().
* Add unit tests for the receiver's context-first and login-fallback target
selection.
The change makes the delivery target deterministic. It does not rebind a client
already created under a different Subject.
h2. Scope and boundaries
h3. Out of scope
* Preventing HDFS client initialization or blocking RPCs from occurring inside
freshUGI.doAs(...) or the provider's ForkJoinPool thread.
* Closing or rebinding FileSystem instances created under a transient Subject;
this is a separate client-lifecycle cleanup.
* Changing the provider's getCurrentUser() usage. The provider already acquires
tokens inside KerberosLoginProvider.doLoginAndReturnUGI(); its remaining
getCurrentUser() use is a token-required pre-check and renewer-name lookup. A
provider change is a separate follow-up if provider-side behavior is observed.
* Double security installation and SecurityUtils.install idempotency. A
diagnostic build where the second installation was skipped still failed at the
24h deadline.
h3. Version context
Flink: before 1.17, the legacy HadoopModule path added credentials directly to
the login user; the HadoopDelegationTokenReceiver / getCurrentUser() path
starts with 1.17. 1.18, 1.19, and 1.20 share that receiver behavior.
Hadoop: Client.getRpcResponse uses CompletableFuture.get() starting with 3.5.0
(HADOOP-19235). 3.3.x and 3.4.x use AsyncGet.Util.wait. The trigger in this
issue requires the CompletableFuture-based wait path; the receiver defect is
independent of the Hadoop version.
h3. Deployment scope
The failure was observed in a Kubernetes application-mode job using the 3.5.0
IPC behavior. It is not a Kubernetes-specific defect: the trigger is the
client's wait path and the receiver's executing Subject, not the deployment
type. Environments using the pre-3.5.0 AsyncGet.Util.wait path do not create
FJP workers from the provider wait itself.
h2. Testing
* Add receiver unit tests for (a) HadoopSecurityContext UGI precedence, (b)
login fallback when no HadoopSecurityContext is installed, and (c) the
inherited current UGI not being targeted.
* A deployed application-mode job using the Hadoop 3.5.0 IPC behavior should
survive the 24h token deadline and continue checkpointing without
SecretManager$InvalidToken. Verify that the checkpoint-storage DFSClient
receives the new token.
* Run a controlled application-mode probe that verifies the receiver updates
the UGI selected by the new logic rather than a pooled thread's inherited
current UGI; cover the TaskManager receiver path separately.
h2. Acceptance criteria
* A token refresh updates the UGI held by HadoopSecurityContext when present,
otherwise the login UGI, and does not use a pooled thread's inherited current
UGI; a checkpoint-storage client bound to the selected UGI receives the new
token.
* A long-running application-mode job using the Hadoop 3.5.0 IPC behavior
survives the 24h HDFS token deadline.
was:
h2. Summary
Flink periodically obtains HDFS delegation tokens and distributes them through
HadoopDelegationTokenReceiver. The receiver adds refreshed credentials to
UserGroupInformation.getCurrentUser() of the thread running the refresh. A
long-lived Hadoop client, especially a checkpoint-storage DFSClient, may
instead use the UGI/Subject captured when that client was created. If those
identities differ, the refresh succeeds and the receiver reports success, but
the new credentials are not delivered to the client's UGI. The client keeps
using its startup token and fails at the 24h HDFS token deadline with
SecretManager$InvalidToken.
The defect is receiver target selection, not token acquisition or client
binding. A pooled thread's current UGI can be inherited from a different
Subject lifecycle than the UGI held by HadoopSecurityContext or the login user.
The proposed fix targets the UGI held by HadoopSecurityContext, falling back to
the login UGI, instead of a pooled thread's current UGI.
Here "UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means
the identity it carries; identity comparisons use Subject identity.
h2. Observed failure
This was observed in a Flink 1.20.5 Kubernetes application-mode job using
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The job
used a Hadoop IPC client with the 3.5.0 CompletableFuture-based wait path. The
primary failure was in the checkpoint storage path:
{noformat}
CheckpointCoordinator.initializeCheckpointLocation
-> FsCheckpointStorageAccess.initializeLocationForCheckpoint
-> HadoopFileSystem.mkdirs
-> DFSClient.mkdirs
{noformat}
After the 24h deadline, the checkpoint-storage client reported InvalidToken for
its startup token even though refresh had obtained a new HDFS token and the
receiver reported success.
Not every application-mode job is affected. The failure is timing-dependent: a
long-lived Hadoop client must capture one Subject/UGI lifecycle, and a later
refresh receiver must run under a different one. The IPC wait path determines
whether the refresh thread can create such a divergent worker.
h2. Root cause
HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() of
the refresh thread. getCurrentUser() resolves from the thread's effective
AccessControlContext and is not guaranteed to be the UGI held by
HadoopSecurityContext or the UGI captured by a long-lived checkpoint-storage
DFSClient.
In the observed failure, the checkpoint client's captured UGI had the
login/HadoopSecurityContext Subject, while the receiver could run under a
pooled thread's inherited Subject.
h3. How the refresh thread's current UGI can differ
The provider obtains tokens inside freshUGI.doAs(...). With Apache Hadoop 3.5.0
and later, Client.getRpcResponse waits on CompletableFuture.get()
(HADOOP-19235). On a ForkJoinWorkerThread, CompletableFuture.get() enters
ForkJoinPool.managedBlock(); if no idle worker is available and the pool has
capacity, ForkJoinPool.tryCompensate() creates a replacement worker while
freshUGI.doAs(...) is still effective.
The Thread constructor copies the effective AccessControlContext into the new
thread's inheritedAccessControlContext, so a worker created during that window
retains the fresh Subject after doAs returns. A later cluster-io worker created
by one of those workers inherits the same Subject. A receiver running on that
cluster-io worker then resolves getCurrentUser() to the fresh Subject and
writes credentials there, while the login and HadoopSecurityContext UGIs remain
unchanged.
{noformat}
freshUGI.doAs(...)
-> fs.addDelegationTokens(...)
-> HDFS RPC in CompletableFuture.get() [Hadoop 3.5.0+]
-> ForkJoinPool.managedBlock()
-> tryCompensate()
-> createWorker()
-> new dispatcher worker inherits the fresh Subject
-> later creates a cluster-io worker
-> receiver runs with the fresh Subject
{noformat}
In Apache Hadoop 3.3.x and 3.4.x, the same call waits in AsyncGet.Util.wait
(Object.wait) instead. Object.wait is not an FJP managed block, so the pool
does not create a replacement worker; the provider thread blocks without
propagating the fresh Subject to a new thread. This version boundary explains
why the failure is not reproducible with those clients, but it does not make
the receiver's getCurrentUser() assumption correct.
h2. Proposed change
This issue fixes token delivery for the observed long-lived-client case,
primarily checkpoint storage.
* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, select a single target UGI: the UGI held by
HadoopSecurityContext when present; otherwise fall back to
UserGroupInformation.getLoginUser(). Add refreshed credentials only to that
UGI. Do not target UserGroupInformation.getCurrentUser().
* Add unit tests for the receiver's context-first and login-fallback target
selection.
The change makes the delivery target deterministic. It does not rebind a client
already created under a different Subject.
h2. Scope and boundaries
h3. Out of scope
* Preventing HDFS client initialization or blocking RPCs from occurring inside
freshUGI.doAs(...) or the provider's ForkJoinPool thread.
* Closing or rebinding FileSystem instances created under a transient Subject;
this is a separate client-lifecycle cleanup.
* Changing the provider's getCurrentUser() usage. The provider already acquires
tokens inside KerberosLoginProvider.doLoginAndReturnUGI(); its remaining
getCurrentUser() use is a token-required pre-check and renewer-name lookup. A
provider change is a separate follow-up if provider-side behavior is observed.
* Double security installation and SecurityUtils.install idempotency. A
diagnostic build where the second installation was skipped still failed at the
24h deadline.
h3. Version context
Flink: before 1.17, the legacy HadoopModule path added credentials directly to
the login user; the HadoopDelegationTokenReceiver / getCurrentUser() path
starts with 1.17. 1.18, 1.19, and 1.20 share that receiver behavior.
Hadoop: Client.getRpcResponse uses CompletableFuture.get() starting with 3.5.0
(HADOOP-19235). 3.3.x and 3.4.x use AsyncGet.Util.wait. The trigger in this
issue requires the CompletableFuture-based wait path; the receiver defect is
independent of the Hadoop version.
h3. Deployment scope
The failure was observed in a Kubernetes application-mode job using the 3.5.0
IPC behavior. It is not a Kubernetes-specific defect: the trigger is the
client's wait path and the receiver's executing Subject, not the deployment
type. Environments using the pre-3.5.0 AsyncGet.Util.wait path do not create
FJP workers during the provider window.
h2. Testing
* Add receiver unit tests for (a) HadoopSecurityContext UGI precedence, (b)
login fallback when no HadoopSecurityContext is installed, and (c) the
inherited current UGI not being targeted.
* A deployed application-mode job using the Hadoop 3.5.0 IPC behavior should
survive the 24h token deadline and continue checkpointing without
SecretManager$InvalidToken. Verify that the checkpoint-storage DFSClient
receives the new token.
* Run a controlled application-mode probe that verifies the receiver updates
the UGI selected by the new logic rather than a pooled thread's inherited
current UGI; cover the TaskManager receiver path separately.
h2. Acceptance criteria
* A token refresh updates the UGI held by HadoopSecurityContext when present,
otherwise the login UGI, and does not use a pooled thread's inherited current
UGI; a checkpoint-storage client bound to the selected UGI receives the new
token.
* A long-running application-mode job using the Hadoop 3.5.0 IPC behavior
survives the 24h HDFS token deadline.
> Hadoop delegation token refresh can target a different UGI than Hadoop clients
> ------------------------------------------------------------------------------
>
> Key: FLINK-41435
> URL: https://issues.apache.org/jira/browse/FLINK-41435
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
> Affects Versions: 2.0.0, 1.19.0, 1.20.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0
> Environment: Flink 1.19.0 and later (1.19.x, 1.20.x, 2.0 through
> 2.4), plus master (2.5-SNAPSHOT); Kubernetes and standalone application mode
> with Kerberos and HDFS delegation tokens. The mismatch is more likely when
> FileSystemJobResultStore and the token manager share the cluster-io executor
> (FLINK-27204).
> Reporter: LuChengxiang
> Priority: Major
>
> h2. Summary
> Flink periodically obtains HDFS delegation tokens and distributes them
> through HadoopDelegationTokenReceiver. The receiver adds refreshed
> credentials to UserGroupInformation.getCurrentUser() of the thread running
> the refresh. A long-lived Hadoop client, especially a checkpoint-storage
> DFSClient, may instead use the UGI/Subject captured when that client was
> created. If those identities differ, the refresh succeeds and the receiver
> reports success, but the new credentials are not delivered to the client's
> UGI. The client keeps using its startup token and fails at the 24h HDFS token
> deadline with SecretManager$InvalidToken.
> The defect is receiver target selection, not token acquisition or client
> binding. A pooled thread's current UGI can be inherited from a different
> Subject lifecycle than the UGI held by HadoopSecurityContext or the login
> user. The proposed fix targets the UGI held by HadoopSecurityContext, falling
> back to the login UGI, instead of a pooled thread's current UGI.
> Here "UGI" means the Hadoop UserGroupInformation wrapper and "Subject" means
> the identity it carries; identity comparisons use Subject identity.
> h2. Observed failure
> This was observed in a Flink 1.20.5 Kubernetes application-mode job using
> ZooKeeper-based high availability and HDFS-backed checkpoint storage. The job
> used a Hadoop IPC client with the 3.5.0 CompletableFuture-based wait path.
> The primary failure was in the checkpoint storage path:
> {noformat}
> CheckpointCoordinator.initializeCheckpointLocation
> -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
> -> HadoopFileSystem.mkdirs
> -> DFSClient.mkdirs
> {noformat}
> After the 24h deadline, the checkpoint-storage client reported InvalidToken
> for its startup token even though refresh had obtained a new HDFS token and
> the receiver reported success.
> Not every application-mode job is affected. The failure is timing-dependent:
> a long-lived Hadoop client must capture one Subject/UGI lifecycle, and a
> later refresh receiver must run under a different one. The IPC wait path
> determines whether the refresh thread can create such a divergent worker.
> h2. Root cause
> HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser()
> of the refresh thread. getCurrentUser() resolves from the thread's effective
> AccessControlContext and is not guaranteed to be the UGI held by
> HadoopSecurityContext or the UGI captured by a long-lived checkpoint-storage
> DFSClient.
> In the observed failure, the checkpoint client's captured UGI had the
> login/HadoopSecurityContext Subject, while the receiver could run under a
> pooled thread's inherited Subject.
> h3. How the refresh thread's current UGI can differ
> The provider obtains tokens inside freshUGI.doAs(...). With Apache Hadoop
> 3.5.0 and later, Client.getRpcResponse waits on CompletableFuture.get()
> (HADOOP-19235). On a ForkJoinWorkerThread, CompletableFuture.get() enters
> ForkJoinPool.managedBlock(); if no idle worker is available and the pool has
> capacity, ForkJoinPool.tryCompensate() creates a replacement worker while
> freshUGI.doAs(...) is still effective.
> The Thread constructor copies the effective AccessControlContext into the new
> thread's inheritedAccessControlContext, so a worker created during that
> window retains the fresh Subject after doAs returns. A later cluster-io
> worker created by one of those workers inherits the same Subject. A receiver
> running on that cluster-io worker then resolves getCurrentUser() to the fresh
> Subject and writes credentials there, while the login and
> HadoopSecurityContext UGIs remain unchanged.
> {noformat}
> freshUGI.doAs(...)
> -> fs.addDelegationTokens(...)
> -> HDFS RPC in CompletableFuture.get() [Hadoop 3.5.0+]
> -> ForkJoinPool.managedBlock()
> -> tryCompensate()
> -> createWorker()
> -> new dispatcher worker inherits the fresh Subject
> -> later creates a cluster-io worker
> -> receiver runs with the fresh Subject
> {noformat}
> In Apache Hadoop 3.3.x and 3.4.x, the same call waits in AsyncGet.Util.wait
> (Object.wait) instead. Object.wait is not an FJP managed block, so the pool
> does not create a replacement worker; the provider thread blocks without
> propagating the fresh Subject to a new thread. This version boundary explains
> why the failure is not reproducible with those clients, but it does not make
> the receiver's getCurrentUser() assumption correct.
> h2. Proposed change
> This issue fixes token delivery for the observed long-lived-client case,
> primarily checkpoint storage.
> * Add an accessor for the UGI held by HadoopSecurityContext.
> * In HadoopDelegationTokenReceiver, select a single target UGI: the UGI held
> by HadoopSecurityContext when present; otherwise fall back to
> UserGroupInformation.getLoginUser(). Add refreshed credentials only to that
> UGI. Do not target UserGroupInformation.getCurrentUser().
> * Add unit tests for the receiver's context-first and login-fallback target
> selection.
> The change makes the delivery target deterministic. It does not rebind a
> client already created under a different Subject.
> h2. Scope and boundaries
> h3. Out of scope
> * Preventing HDFS client initialization or blocking RPCs from occurring
> inside freshUGI.doAs(...) or the provider's ForkJoinPool thread.
> * Closing or rebinding FileSystem instances created under a transient
> Subject; this is a separate client-lifecycle cleanup.
> * Changing the provider's getCurrentUser() usage. The provider already
> acquires tokens inside KerberosLoginProvider.doLoginAndReturnUGI(); its
> remaining getCurrentUser() use is a token-required pre-check and renewer-name
> lookup. A provider change is a separate follow-up if provider-side behavior
> is observed.
> * Double security installation and SecurityUtils.install idempotency. A
> diagnostic build where the second installation was skipped still failed at
> the 24h deadline.
> h3. Version context
> Flink: before 1.17, the legacy HadoopModule path added credentials directly
> to the login user; the HadoopDelegationTokenReceiver / getCurrentUser() path
> starts with 1.17. 1.18, 1.19, and 1.20 share that receiver behavior.
> Hadoop: Client.getRpcResponse uses CompletableFuture.get() starting with
> 3.5.0 (HADOOP-19235). 3.3.x and 3.4.x use AsyncGet.Util.wait. The trigger in
> this issue requires the CompletableFuture-based wait path; the receiver
> defect is independent of the Hadoop version.
> h3. Deployment scope
> The failure was observed in a Kubernetes application-mode job using the 3.5.0
> IPC behavior. It is not a Kubernetes-specific defect: the trigger is the
> client's wait path and the receiver's executing Subject, not the deployment
> type. Environments using the pre-3.5.0 AsyncGet.Util.wait path do not create
> FJP workers from the provider wait itself.
> h2. Testing
> * Add receiver unit tests for (a) HadoopSecurityContext UGI precedence, (b)
> login fallback when no HadoopSecurityContext is installed, and (c) the
> inherited current UGI not being targeted.
> * A deployed application-mode job using the Hadoop 3.5.0 IPC behavior should
> survive the 24h token deadline and continue checkpointing without
> SecretManager$InvalidToken. Verify that the checkpoint-storage DFSClient
> receives the new token.
> * Run a controlled application-mode probe that verifies the receiver updates
> the UGI selected by the new logic rather than a pooled thread's inherited
> current UGI; cover the TaskManager receiver path separately.
> h2. Acceptance criteria
> * A token refresh updates the UGI held by HadoopSecurityContext when present,
> otherwise the login UGI, and does not use a pooled thread's inherited current
> UGI; a checkpoint-storage client bound to the selected UGI receives the new
> token.
> * A long-running application-mode job using the Hadoop 3.5.0 IPC behavior
> survives the 24h HDFS token deadline.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)