[
https://issues.apache.org/jira/browse/FLINK-41435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LuChengxiang updated FLINK-41435:
---------------------------------
Description:
h2. Summary
Flink periodically obtains HDFS delegation tokens and distributes them through
HadoopDelegationTokenReceiver. The receiver currently adds new credentials to
UserGroupInformation.getCurrentUser() on the thread that performs the refresh.
Long-lived Hadoop clients, especially checkpoint storage, can capture a
UGI/Subject when their DFSClient is created and use that captured identity for
later RPC connections.
If the refresh path resolves to a different current UGI from the one captured
by a long-lived client, the client can continue using its startup token even
though a new token was obtained successfully. The failure appears at the 24h
HDFS token deadline as SecretManager$InvalidToken.
The observed Kubernetes reproduction is in the initial token acquisition path.
The receiver mismatch is not limited to that path: it applies whenever a
refresh task resolves to a different current UGI. In this issue, "UGI" means
the Hadoop UserGroupInformation wrapper, while "Subject" means the identity
carried by that UGI; identity comparisons between wrappers use Subject identity.
h2. Observed failure
This was observed in a Flink 1.20.5 Kubernetes application-mode job using
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The
primary failure was in the checkpoint storage path:
{noformat}
CheckpointCoordinator.initializeCheckpointLocation
-> FsCheckpointStorageAccess.initializeLocationForCheckpoint
-> HadoopFileSystem.mkdirs
-> DFSClient.mkdirs
{noformat}
The checkpoint storage client reported InvalidToken for the startup token after
the refresh had created a new HDFS token and the receiver reported success. The
same JVM also showed a FileSystemJobResultStore stack using the expired token
during cleanup. The job-result store is a secondary long-lived client with the
same failure mode, not the primary trigger.
Not every Kubernetes application-mode job is affected. The failure is
timing-dependent: it requires a long-lived Hadoop client to have captured one
Subject/UGI lifecycle while the refresh path later resolves to a different one.
Installation idempotency is out of scope. A diagnostic build that skipped the
second security installation still failed at the 24h HDFS token deadline. The
Kubernetes application entrypoint (1.19+) installs security early for artifact
fetching and then installs again in {{startCluster}}; this creates an
additional security-module/context lifecycle, but the probe does not establish
it as a cause of the receiver mismatch. Flink's {{FileSystem.initialize}}
clears only Flink's {{FileSystem}} cache; it does not clear Hadoop
{{FileSystem}}/{{DFSClient}} caches or rebind existing UGI/Subject-held clients.
h2. Root cause
HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() of
the thread executing the refresh. That current UGI is resolved from the
thread's effective AccessControlContext and is not guaranteed to be the UGI
held by HadoopSecurityContext or the UGI captured by a long-lived
checkpoint-storage or job-result-store DFSClient.
The controlled Kubernetes probe observed the initial token acquisition path
({{ResourceManager.onStart}} -> {{DefaultDelegationTokenManager.start()}} ->
{{startTokensUpdate()}}), not a later scheduled refresh; the later scheduled
refresh is submitted through the cluster-io executor. The startup FJP expansion
is observed; the same mechanism is possible for any provider {{doAs}} on a
ForkJoinWorkerThread with a blocking HDFS RPC, but that later path has not been
directly reproduced in this probe.
{noformat}
provider thread = default-dispatcher-12
freshUGI Subject = 817147333
login/HadoopSecurityContext Subject = 1333938290
current Subject before doAs = 1333938290
current Subject inside doAs = 817147333
current Subject after doAs = 1333938290
FJP workers created during the provider's blocked RPC:
default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12
later cluster-io worker:
cluster-io-thread-3, creator=default-dispatcher-15
current Subject = 817147333
login/HadoopSecurityContext Subject = 1333938290
JM-local receiver target (getCurrentUser()) = 817147333
{noformat}
Here {{HadoopSecurityContext Subject}} means the Subject of the UGI held by the
installed {{HadoopSecurityContext}}. In this probe it is the same Subject
identity as the login user; that is not guaranteed in general.
The fresh Subject is not created by HadoopSecurityContext or by the dispatcher.
It belongs to the UGI returned by
{{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls
{{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}.
That call performs a Kerberos login and returns a newly constructed UGI for the
same principal; it does not replace the process login user.
During {{freshUGI.doAs(...)}}, {{fs.addDelegationTokens(...)}} performed an
HDFS delegation-token RPC and blocked in {{CompletableFuture.get()}}. Because
the caller was a ForkJoinWorkerThread, ForkJoinPool entered its managed-block
compensation path and created new dispatcher workers while the fresh Subject
was active. The Thread constructor captures the {{AccessControlContext}}
snapshot returned by {{AccessController.getContext()}}. That snapshot reflects
the current thread's effective, stack-derived context, including the active
{{Subject.doAs(...)}} Subject; it is not the entire live Java stack and not
merely a reference to one stack frame. The new thread stores this snapshot in
its {{inheritedAccessControlContext}}. After {{doAs}} returned, current on the
creator resolved to the login Subject again, but the new workers retained the
inherited fresh Subject.
A dispatcher worker created during the {{doAs}} window later created
{{cluster-io-thread-3}}. On {{cluster-io-thread-3}}, current Subject was the
fresh Subject, while login and HadoopSecurityContext Subjects remained
{{1333938290}}. If a receiver invocation runs on this worker, the current
JM-local implementation calls
{{UserGroupInformation.getCurrentUser().addCredentials(...)}}. That resolves to
a UGI whose Subject is the fresh Subject, so the credentials are written to
that UGI's Subject rather than to the UGI captured by the long-lived client.
This is a per-thread inherited security context, not a global mutation of
{{UserGroupInformation}}. The probe covers only the JobManager receiver path;
TaskManager receivers receive token updates through the ResourceManager.
h2. YARN status and scope
In a controlled YARN application-mode probe, the provider also ran on a Pekko
default-dispatcher worker. Its provider window stayed at {{poolSize=3}} /
{{active=2}} with no {{tryCompensate}} or {{createWorker}}, so no new
dispatcher worker was created while the provider's fresh Subject was active.
YARN did show ForkJoinPool expansion during the separate JobGraph-generation
startup path, so this is not evidence that YARN is immune.
This is one YARN sample; the provider's {{CompletableFuture}} state was not
instrumented, so the YARN trigger is not established. YARN is not listed as a
confirmed reproduction, but it is not claimed to be exempt.
h2. Version context
The receiver has used {{UserGroupInformation.getCurrentUser()}} since 1.17;
1.18, 1.19, and 1.20 share that receiver behavior. The observed primary failure
is the checkpoint-storage DFSClient; {{FileSystemJobResultStore}} is a second
long-lived client with the same target mismatch. The version-specific code
difference is that 1.18 uses synchronous {{FileSystemJobResultStore}} I/O,
while 1.19+ moves that I/O onto the shared {{cluster-io}} executor
({{FLINK-27204}}). That change creates a concrete pooled-worker
Subject-propagation path; it is not the only client path, and checkpoint
storage exists in both versions. The Kubernetes application entrypoint also
installs security early for artifact fetching and then installs again in
{{startCluster}} in 1.19/1.20. These are exposure-path differences, not
separate root causes; 1.18 is not claimed to be immune.
h2. Proposed change
This issue addresses the case where a token is obtained successfully but the
refreshed credentials do not reach long-lived Hadoop clients, primarily
checkpoint storage.
* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, select a single target UGI: use the UGI
held by HadoopSecurityContext when present; otherwise fall back to
{{UserGroupInformation.getLoginUser()}}. Add refreshed credentials only to that
UGI. Do not target {{UserGroupInformation.getCurrentUser()}}, which may be a
pooled worker's inherited Subject.
* Add tests covering (a) the HadoopSecurityContext UGI takes precedence over
login/current, (b) the login UGI is used when no HadoopSecurityContext is
installed, and (c) the inherited current UGI is not updated.
Out of scope for this issue:
* Preventing HDFS client initialization or blocking RPCs from occurring inside
{{freshUGI.doAs(...)}} or the provider's ForkJoinPool worker.
* Closing or rebinding FileSystem instances created under a transient fresh
Subject; this is a separate client-lifecycle cleanup.
* Delivering credentials to clients created under an unrelated transient
Subject; those clients require separate rebinding or recreation.
* Changing the provider's {{getCurrentUser()}} usage.
{{HadoopFSDelegationTokenProvider}} already acquires tokens inside
{{KerberosLoginProvider.doLoginAndReturnUGI()}}; its remaining
{{getCurrentUser()}} use is a token-required pre-check and renewer-name lookup.
No provider change is required for the observed receiver mismatch; it remains a
separate follow-up if provider-side behavior is observed.
* SecurityUtils.install idempotency.
The proposed receiver change does not rely solely on the refresh thread's
inherited Subject. It does not by itself guarantee that a client already
created under a different Subject is rebound. That remains a separate lifecycle
concern.
h2. Testing
* A controlled Kubernetes application-mode probe recorded the chain from
{{fs.addDelegationTokens()}} through Hadoop ipc {{Client.getRpcResponse()}},
{{CompletableFuture.get()}}, {{ForkJoinPool.managedBlock()}},
{{tryCompensate()}}, and Pekko {{ThreadFactory.newThread()}}. The creator and
the new dispatcher workers had the same fresh Subject identity.
* The same probe recorded {{cluster-io-thread-3}} created by
{{default-dispatcher-15}} with {{subjectId=817147333}}, while the
login/HadoopSecurityContext Subject remained {{1333938290}}.
* A JM-local receiver invocation on {{cluster-io-thread-3}} observed current
Subject=817147333 while login/HadoopSecurityContext Subject=1333938290. The
current receiver therefore targets the fresh UGI on this worker; TaskManager
receiver behavior is not covered by this probe and requires a separate
TaskManager receiver test.
* Add receiver unit tests for (a) HadoopSecurityContext UGI precedence with
login/current not updated, (b) login fallback when no HadoopSecurityContext is
installed, and (c) the inherited current UGI not targeted.
* Trigger multiple post-startup token refreshes in a long-running job and
verify which executor runs the provider, whether new ForkJoinPool workers are
created, and which Subject the receiver's current user resolves to.
* Repeat the YARN provider-window probe while instrumenting the provider's
{{CompletableFuture}} state; the current YARN sample is summarized in "YARN
status and scope".
* A deployed application-mode job should survive the 24h token deadline and
continue checkpointing without SecretManager$InvalidToken. Verify that the
checkpoint-storage DFSClient receives the new token, and verify the
job-result-store client separately because it can capture a different UGI.
h2. Acceptance criteria
* A token refresh updates the UGI held by HadoopSecurityContext when present,
otherwise the login UGI, and does not use a pooled worker's inherited current
UGI; a checkpoint-storage client bound to the selected UGI receives the new
token.
* A long-running application-mode job survives the 24h HDFS token deadline.
was:
h2. Summary
Flink periodically obtains HDFS delegation tokens and distributes them through
HadoopDelegationTokenReceiver. The receiver currently adds new credentials to
UserGroupInformation.getCurrentUser() on the thread that performs the refresh.
Long-lived Hadoop clients, especially checkpoint storage, can capture a
UGI/Subject when their DFSClient is created and use that captured identity for
later RPC connections.
If the refresh path resolves to a different current UGI from the one captured
by a long-lived client, the client can continue using its startup token even
though a new token was obtained successfully. The failure appears at the 24h
HDFS token deadline as SecretManager$InvalidToken.
The observed Kubernetes reproduction is in the initial token acquisition path.
The receiver mismatch is not limited to that path: it applies whenever a
refresh task resolves to a different current UGI. In this issue, "UGI" means
the Hadoop UserGroupInformation wrapper, while "Subject" means the identity
carried by that UGI; identity comparisons between wrappers use Subject identity.
h2. Observed failure
This was observed in a Flink 1.20.5 Kubernetes application-mode job using
ZooKeeper-based high availability and HDFS-backed checkpoint storage. The
primary failure was in the checkpoint storage path:
{noformat}
CheckpointCoordinator.initializeCheckpointLocation
-> FsCheckpointStorageAccess.initializeLocationForCheckpoint
-> HadoopFileSystem.mkdirs
-> DFSClient.mkdirs
{noformat}
The checkpoint storage client reported InvalidToken for the startup token after
the refresh had created a new HDFS token and the receiver reported success. The
same JVM also showed a FileSystemJobResultStore stack using the expired token
during cleanup. The job-result store is a secondary long-lived client with the
same failure mode, not the primary trigger.
Not every Kubernetes application-mode job is affected. The failure is
timing-dependent: it requires a long-lived Hadoop client to have captured one
Subject/UGI lifecycle while the refresh path later resolves to a different one.
Installation idempotency is out of scope. A diagnostic build that skipped the
second security installation still failed at the 24h HDFS token deadline. The
Kubernetes application entrypoint (1.19+) installs security early for artifact
fetching and then installs again in {{startCluster}}; this creates an
additional security-module/context lifecycle, but the probe does not establish
it as a cause of the receiver mismatch. Flink's {{FileSystem.initialize}}
clears only Flink's {{FileSystem}} cache; it does not clear Hadoop
{{FileSystem}}/{{DFSClient}} caches or rebind existing UGI/Subject-held clients.
h2. Root cause
HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser() of
the thread executing the refresh. That current UGI is resolved from the
thread's effective AccessControlContext and is not guaranteed to be the UGI
held by HadoopSecurityContext or the UGI captured by a long-lived
checkpoint-storage or job-result-store DFSClient.
The controlled Kubernetes probe observed the initial token acquisition path
({{ResourceManager.onStart}} -> {{DefaultDelegationTokenManager.start()}} ->
{{startTokensUpdate()}}), not a later scheduled refresh; the later scheduled
refresh is submitted through the cluster-io executor. The startup FJP expansion
is observed; the same mechanism is possible for any provider {{doAs}} on a
ForkJoinWorkerThread with a blocking HDFS RPC, but that later path has not been
directly reproduced in this probe.
{noformat}
provider thread = default-dispatcher-12
freshUGI Subject = 817147333
login/HadoopSecurityContext Subject = 1333938290
current Subject before doAs = 1333938290
current Subject inside doAs = 817147333
current Subject after doAs = 1333938290
FJP workers created during the provider's blocked RPC:
default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12
later cluster-io worker:
cluster-io-thread-3, creator=default-dispatcher-15
current Subject = 817147333
login/HadoopSecurityContext Subject = 1333938290
JM-local receiver target (getCurrentUser()) = 817147333
{noformat}
Here {{HadoopSecurityContext Subject}} means the Subject of the UGI held by the
installed {{HadoopSecurityContext}}. In this probe it is the same Subject
identity as the login user; that is not guaranteed in general.
The fresh Subject is not created by HadoopSecurityContext or by the dispatcher.
It belongs to the UGI returned by
{{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls
{{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}.
That call performs a Kerberos login and returns a newly constructed UGI for the
same principal; it does not replace the process login user.
During {{freshUGI.doAs(...)}}, {{fs.addDelegationTokens(...)}} performed an
HDFS delegation-token RPC and blocked in {{CompletableFuture.get()}}. Because
the caller was a ForkJoinWorkerThread, ForkJoinPool entered its managed-block
compensation path and created new dispatcher workers while the fresh Subject
was active. The Thread constructor captures the {{AccessControlContext}}
snapshot returned by {{AccessController.getContext()}}. That snapshot reflects
the current thread's effective, stack-derived context, including the active
{{Subject.doAs(...)}} Subject; it is not the entire live Java stack and not
merely a reference to one stack frame. The new thread stores this snapshot in
its {{inheritedAccessControlContext}}. After {{doAs}} returned, current on the
creator resolved to the login Subject again, but the new workers retained the
inherited fresh Subject.
A dispatcher worker created during the {{doAs}} window later created
{{cluster-io-thread-3}}. On {{cluster-io-thread-3}}, current Subject was the
fresh Subject, while login and HadoopSecurityContext Subjects remained
{{1333938290}}. If a receiver invocation runs on this worker, the current
JM-local implementation calls
{{UserGroupInformation.getCurrentUser().addCredentials(...)}}. That resolves to
a UGI whose Subject is the fresh Subject, so the credentials are written to
that UGI's Subject rather than to the UGI captured by the long-lived client.
This is a per-thread inherited security context, not a global mutation of
{{UserGroupInformation}}. The probe covers only the JobManager receiver path;
TaskManager receivers receive token updates through the ResourceManager.
h2. YARN status and scope
In a controlled YARN application-mode probe, the provider also ran on a Pekko
default-dispatcher worker. Its provider window stayed at {{poolSize=3}} /
{{active=2}} with no {{tryCompensate}} or {{createWorker}}, so no new
dispatcher worker was created while the provider's fresh Subject was active.
YARN did show ForkJoinPool expansion during the separate JobGraph-generation
startup path, so this is not evidence that YARN is immune.
This is one YARN sample; the provider's {{CompletableFuture}} state was not
instrumented, so the YARN trigger is not established. YARN is not listed as a
confirmed reproduction, but it is not claimed to be exempt.
h2. Version context
The receiver has used {{UserGroupInformation.getCurrentUser()}} since 1.17;
1.18, 1.19, and 1.20 share that receiver behavior. The exposure differs around
it: 1.18 uses synchronous {{FileSystemJobResultStore}} I/O, while 1.19+ moves
that I/O onto the shared {{cluster-io}} executor ({{FLINK-27204}}). The
Kubernetes application entrypoint also installs security early for artifact
fetching and then installs again in {{startCluster}} in 1.19/1.20. These are
exposure-path differences, not separate root causes; 1.18 is not claimed to be
immune.
h2. Proposed change
This issue addresses the case where a token is obtained successfully but the
refreshed credentials do not reach long-lived Hadoop clients, primarily
checkpoint storage.
* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, select a single target UGI: use the UGI
held by HadoopSecurityContext when present; otherwise fall back to
{{UserGroupInformation.getLoginUser()}}. Add refreshed credentials only to that
UGI. Do not target {{UserGroupInformation.getCurrentUser()}}, which may be a
pooled worker's inherited Subject.
* Add tests covering (a) the HadoopSecurityContext UGI takes precedence over
login/current, (b) the login UGI is used when no HadoopSecurityContext is
installed, and (c) the inherited current UGI is not updated.
Out of scope for this issue:
* Preventing HDFS client initialization or blocking RPCs from occurring inside
{{freshUGI.doAs(...)}} or the provider's ForkJoinPool worker.
* Closing or rebinding FileSystem instances created under a transient fresh
Subject; this is a separate client-lifecycle cleanup.
* Delivering credentials to clients created under an unrelated transient
Subject; those clients require separate rebinding or recreation.
* Changing the provider's {{getCurrentUser()}} usage.
{{HadoopFSDelegationTokenProvider}} already acquires tokens inside
{{KerberosLoginProvider.doLoginAndReturnUGI()}}; its remaining
{{getCurrentUser()}} use is a token-required pre-check and renewer-name lookup.
No provider change is required for the observed receiver mismatch; it remains a
separate follow-up if provider-side behavior is observed.
* SecurityUtils.install idempotency.
The proposed receiver change does not rely solely on the refresh thread's
inherited Subject. It does not by itself guarantee that a client already
created under a different Subject is rebound. That remains a separate lifecycle
concern.
h2. Testing
* A controlled Kubernetes application-mode probe recorded the chain from
{{fs.addDelegationTokens()}} through Hadoop ipc {{Client.getRpcResponse()}},
{{CompletableFuture.get()}}, {{ForkJoinPool.managedBlock()}},
{{tryCompensate()}}, and Pekko {{ThreadFactory.newThread()}}. The creator and
the new dispatcher workers had the same fresh Subject identity.
* The same probe recorded {{cluster-io-thread-3}} created by
{{default-dispatcher-15}} with {{subjectId=817147333}}, while the
login/HadoopSecurityContext Subject remained {{1333938290}}.
* A JM-local receiver invocation on {{cluster-io-thread-3}} observed current
Subject=817147333 while login/HadoopSecurityContext Subject=1333938290. The
current receiver therefore targets the fresh UGI on this worker; TaskManager
receiver behavior is not covered by this probe and requires a separate
TaskManager receiver test.
* Add receiver unit tests for (a) HadoopSecurityContext UGI precedence with
login/current not updated, (b) login fallback when no HadoopSecurityContext is
installed, and (c) the inherited current UGI not targeted.
* Trigger multiple post-startup token refreshes in a long-running job and
verify which executor runs the provider, whether new ForkJoinPool workers are
created, and which Subject the receiver's current user resolves to.
* Repeat the YARN provider-window probe while instrumenting the provider's
{{CompletableFuture}} state; the current YARN sample is summarized in "YARN
status and scope".
* A deployed application-mode job should survive the 24h token deadline and
continue checkpointing without SecretManager$InvalidToken. Verify that the
checkpoint-storage DFSClient receives the new token, and verify the
job-result-store client separately because it can capture a different UGI.
h2. Acceptance criteria
* A token refresh updates the UGI held by HadoopSecurityContext when present,
otherwise the login UGI, and does not use a pooled worker's inherited current
UGI; a checkpoint-storage client bound to the selected UGI receives the new
token.
* A long-running application-mode job survives the 24h HDFS token deadline.
> Hadoop delegation token refresh can target a different UGI than Hadoop clients
> ------------------------------------------------------------------------------
>
> Key: FLINK-41435
> URL: https://issues.apache.org/jira/browse/FLINK-41435
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
> Affects Versions: 2.0.0, 1.19.0, 1.20.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0
> Environment: Flink 1.19.0 and later (1.19.x, 1.20.x, 2.0 through
> 2.4), plus master (2.5-SNAPSHOT); Kubernetes and standalone application mode
> with Kerberos and HDFS delegation tokens. The mismatch is more likely when
> FileSystemJobResultStore and the token manager share the cluster-io executor
> (FLINK-27204).
> Reporter: LuChengxiang
> Priority: Major
>
> h2. Summary
> Flink periodically obtains HDFS delegation tokens and distributes them
> through HadoopDelegationTokenReceiver. The receiver currently adds new
> credentials to UserGroupInformation.getCurrentUser() on the thread that
> performs the refresh. Long-lived Hadoop clients, especially checkpoint
> storage, can capture a UGI/Subject when their DFSClient is created and use
> that captured identity for later RPC connections.
> If the refresh path resolves to a different current UGI from the one captured
> by a long-lived client, the client can continue using its startup token even
> though a new token was obtained successfully. The failure appears at the 24h
> HDFS token deadline as SecretManager$InvalidToken.
> The observed Kubernetes reproduction is in the initial token acquisition
> path. The receiver mismatch is not limited to that path: it applies whenever
> a refresh task resolves to a different current UGI. In this issue, "UGI"
> means the Hadoop UserGroupInformation wrapper, while "Subject" means the
> identity carried by that UGI; identity comparisons between wrappers use
> Subject identity.
> h2. Observed failure
> This was observed in a Flink 1.20.5 Kubernetes application-mode job using
> ZooKeeper-based high availability and HDFS-backed checkpoint storage. The
> primary failure was in the checkpoint storage path:
> {noformat}
> CheckpointCoordinator.initializeCheckpointLocation
> -> FsCheckpointStorageAccess.initializeLocationForCheckpoint
> -> HadoopFileSystem.mkdirs
> -> DFSClient.mkdirs
> {noformat}
> The checkpoint storage client reported InvalidToken for the startup token
> after the refresh had created a new HDFS token and the receiver reported
> success. The same JVM also showed a FileSystemJobResultStore stack using the
> expired token during cleanup. The job-result store is a secondary long-lived
> client with the same failure mode, not the primary trigger.
> Not every Kubernetes application-mode job is affected. The failure is
> timing-dependent: it requires a long-lived Hadoop client to have captured one
> Subject/UGI lifecycle while the refresh path later resolves to a different
> one.
> Installation idempotency is out of scope. A diagnostic build that skipped the
> second security installation still failed at the 24h HDFS token deadline. The
> Kubernetes application entrypoint (1.19+) installs security early for
> artifact fetching and then installs again in {{startCluster}}; this creates
> an additional security-module/context lifecycle, but the probe does not
> establish it as a cause of the receiver mismatch. Flink's
> {{FileSystem.initialize}} clears only Flink's {{FileSystem}} cache; it does
> not clear Hadoop {{FileSystem}}/{{DFSClient}} caches or rebind existing
> UGI/Subject-held clients.
> h2. Root cause
> HadoopDelegationTokenReceiver targets UserGroupInformation.getCurrentUser()
> of the thread executing the refresh. That current UGI is resolved from the
> thread's effective AccessControlContext and is not guaranteed to be the UGI
> held by HadoopSecurityContext or the UGI captured by a long-lived
> checkpoint-storage or job-result-store DFSClient.
> The controlled Kubernetes probe observed the initial token acquisition path
> ({{ResourceManager.onStart}} -> {{DefaultDelegationTokenManager.start()}} ->
> {{startTokensUpdate()}}), not a later scheduled refresh; the later scheduled
> refresh is submitted through the cluster-io executor. The startup FJP
> expansion is observed; the same mechanism is possible for any provider
> {{doAs}} on a ForkJoinWorkerThread with a blocking HDFS RPC, but that later
> path has not been directly reproduced in this probe.
> {noformat}
> provider thread = default-dispatcher-12
> freshUGI Subject = 817147333
> login/HadoopSecurityContext Subject = 1333938290
> current Subject before doAs = 1333938290
> current Subject inside doAs = 817147333
> current Subject after doAs = 1333938290
> FJP workers created during the provider's blocked RPC:
> default-dispatcher-13 / 14 / 15, creator=default-dispatcher-12
> later cluster-io worker:
> cluster-io-thread-3, creator=default-dispatcher-15
> current Subject = 817147333
> login/HadoopSecurityContext Subject = 1333938290
> JM-local receiver target (getCurrentUser()) = 817147333
> {noformat}
> Here {{HadoopSecurityContext Subject}} means the Subject of the UGI held by
> the installed {{HadoopSecurityContext}}. In this probe it is the same Subject
> identity as the login user; that is not guaranteed in general.
> The fresh Subject is not created by HadoopSecurityContext or by the
> dispatcher. It belongs to the UGI returned by
> {{KerberosLoginProvider.doLoginAndReturnUGI()}}, which calls
> {{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}.
> That call performs a Kerberos login and returns a newly constructed UGI for
> the same principal; it does not replace the process login user.
> During {{freshUGI.doAs(...)}}, {{fs.addDelegationTokens(...)}} performed an
> HDFS delegation-token RPC and blocked in {{CompletableFuture.get()}}. Because
> the caller was a ForkJoinWorkerThread, ForkJoinPool entered its managed-block
> compensation path and created new dispatcher workers while the fresh Subject
> was active. The Thread constructor captures the {{AccessControlContext}}
> snapshot returned by {{AccessController.getContext()}}. That snapshot
> reflects the current thread's effective, stack-derived context, including the
> active {{Subject.doAs(...)}} Subject; it is not the entire live Java stack
> and not merely a reference to one stack frame. The new thread stores this
> snapshot in its {{inheritedAccessControlContext}}. After {{doAs}} returned,
> current on the creator resolved to the login Subject again, but the new
> workers retained the inherited fresh Subject.
> A dispatcher worker created during the {{doAs}} window later created
> {{cluster-io-thread-3}}. On {{cluster-io-thread-3}}, current Subject was the
> fresh Subject, while login and HadoopSecurityContext Subjects remained
> {{1333938290}}. If a receiver invocation runs on this worker, the current
> JM-local implementation calls
> {{UserGroupInformation.getCurrentUser().addCredentials(...)}}. That resolves
> to a UGI whose Subject is the fresh Subject, so the credentials are written
> to that UGI's Subject rather than to the UGI captured by the long-lived
> client. This is a per-thread inherited security context, not a global
> mutation of {{UserGroupInformation}}. The probe covers only the JobManager
> receiver path; TaskManager receivers receive token updates through the
> ResourceManager.
> h2. YARN status and scope
> In a controlled YARN application-mode probe, the provider also ran on a Pekko
> default-dispatcher worker. Its provider window stayed at {{poolSize=3}} /
> {{active=2}} with no {{tryCompensate}} or {{createWorker}}, so no new
> dispatcher worker was created while the provider's fresh Subject was active.
> YARN did show ForkJoinPool expansion during the separate JobGraph-generation
> startup path, so this is not evidence that YARN is immune.
> This is one YARN sample; the provider's {{CompletableFuture}} state was not
> instrumented, so the YARN trigger is not established. YARN is not listed as a
> confirmed reproduction, but it is not claimed to be exempt.
> h2. Version context
> The receiver has used {{UserGroupInformation.getCurrentUser()}} since 1.17;
> 1.18, 1.19, and 1.20 share that receiver behavior. The observed primary
> failure is the checkpoint-storage DFSClient; {{FileSystemJobResultStore}} is
> a second long-lived client with the same target mismatch. The
> version-specific code difference is that 1.18 uses synchronous
> {{FileSystemJobResultStore}} I/O, while 1.19+ moves that I/O onto the shared
> {{cluster-io}} executor ({{FLINK-27204}}). That change creates a concrete
> pooled-worker Subject-propagation path; it is not the only client path, and
> checkpoint storage exists in both versions. The Kubernetes application
> entrypoint also installs security early for artifact fetching and then
> installs again in {{startCluster}} in 1.19/1.20. These are exposure-path
> differences, not separate root causes; 1.18 is not claimed to be immune.
> h2. Proposed change
> This issue addresses the case where a token is obtained successfully but the
> refreshed credentials do not reach long-lived Hadoop clients, primarily
> checkpoint storage.
> * Add an accessor for the UGI held by HadoopSecurityContext.
> * In HadoopDelegationTokenReceiver, select a single target UGI: use the UGI
> held by HadoopSecurityContext when present; otherwise fall back to
> {{UserGroupInformation.getLoginUser()}}. Add refreshed credentials only to
> that UGI. Do not target {{UserGroupInformation.getCurrentUser()}}, which may
> be a pooled worker's inherited Subject.
> * Add tests covering (a) the HadoopSecurityContext UGI takes precedence over
> login/current, (b) the login UGI is used when no HadoopSecurityContext is
> installed, and (c) the inherited current UGI is not updated.
> Out of scope for this issue:
> * Preventing HDFS client initialization or blocking RPCs from occurring
> inside {{freshUGI.doAs(...)}} or the provider's ForkJoinPool worker.
> * Closing or rebinding FileSystem instances created under a transient fresh
> Subject; this is a separate client-lifecycle cleanup.
> * Delivering credentials to clients created under an unrelated transient
> Subject; those clients require separate rebinding or recreation.
> * Changing the provider's {{getCurrentUser()}} usage.
> {{HadoopFSDelegationTokenProvider}} already acquires tokens inside
> {{KerberosLoginProvider.doLoginAndReturnUGI()}}; its remaining
> {{getCurrentUser()}} use is a token-required pre-check and renewer-name
> lookup. No provider change is required for the observed receiver mismatch; it
> remains a separate follow-up if provider-side behavior is observed.
> * SecurityUtils.install idempotency.
> The proposed receiver change does not rely solely on the refresh thread's
> inherited Subject. It does not by itself guarantee that a client already
> created under a different Subject is rebound. That remains a separate
> lifecycle concern.
> h2. Testing
> * A controlled Kubernetes application-mode probe recorded the chain from
> {{fs.addDelegationTokens()}} through Hadoop ipc {{Client.getRpcResponse()}},
> {{CompletableFuture.get()}}, {{ForkJoinPool.managedBlock()}},
> {{tryCompensate()}}, and Pekko {{ThreadFactory.newThread()}}. The creator and
> the new dispatcher workers had the same fresh Subject identity.
> * The same probe recorded {{cluster-io-thread-3}} created by
> {{default-dispatcher-15}} with {{subjectId=817147333}}, while the
> login/HadoopSecurityContext Subject remained {{1333938290}}.
> * A JM-local receiver invocation on {{cluster-io-thread-3}} observed current
> Subject=817147333 while login/HadoopSecurityContext Subject=1333938290. The
> current receiver therefore targets the fresh UGI on this worker; TaskManager
> receiver behavior is not covered by this probe and requires a separate
> TaskManager receiver test.
> * Add receiver unit tests for (a) HadoopSecurityContext UGI precedence with
> login/current not updated, (b) login fallback when no HadoopSecurityContext
> is installed, and (c) the inherited current UGI not targeted.
> * Trigger multiple post-startup token refreshes in a long-running job and
> verify which executor runs the provider, whether new ForkJoinPool workers are
> created, and which Subject the receiver's current user resolves to.
> * Repeat the YARN provider-window probe while instrumenting the provider's
> {{CompletableFuture}} state; the current YARN sample is summarized in "YARN
> status and scope".
> * A deployed application-mode job should survive the 24h token deadline and
> continue checkpointing without SecretManager$InvalidToken. Verify that the
> checkpoint-storage DFSClient receives the new token, and verify the
> job-result-store client separately because it can capture a different UGI.
> h2. Acceptance criteria
> * A token refresh updates the UGI held by HadoopSecurityContext when present,
> otherwise the login UGI, and does not use a pooled worker's inherited current
> UGI; a checkpoint-storage client bound to the selected UGI receives the new
> token.
> * A long-running application-mode job survives the 24h HDFS token deadline.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)