Parth Chandra created SPARK-58438:
-------------------------------------
Summary: HadoopDelegationTokenManager leaks FileSystem cache
entries on every token renewal
Key: SPARK-58438
URL: https://issues.apache.org/jira/browse/SPARK-58438
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.2.0
Reporter: Parth Chandra
{{{}HadoopDelegationTokenManager{}}}'s periodic token-renewal path creates a
fresh
{{UserGroupInformation}} (UGI) on every renewal cycle but never releases the
{{FileSystem}} instances cached for that UGI, so entries accumulate in the
static
{{FileSystem.CACHE}} for the lifetime of the driver JVM.
h4. Details
On each renewal cycle {{updateTokensTask()}} calls {{{}doLogin(){}}}, which
returns a
*new* UGI in the two credential modes:
* keytab: {{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal,
keytab)}}
* ccache: {{UserGroupInformation.getUGIFromTicketCache(ccache, user)}}
Providers invoked inside {{freshUGI.doAs
{ ... }
}} call {{{}FileSystem.get(...){}}}, which
caches {{FileSystem}} instances in {{FileSystem.CACHE}} keyed (in part) by the
UGI.
Cache entries are only evicted by {{FileSystem.close}} /
{{{}FileSystem.closeAllForUGI{}}}.
The renewal path never calls either, so each cycle leaves behind {{FileSystem}}
instances for a throwaway UGI that is never used again.
The one-shot path {{obtainDelegationTokens(creds)}} already cleans up correctly:
{code:scala}
if (!currentUser.equals(freshUGI)) {
FileSystem.closeAllForUGI(freshUGI)
}
{code}
but the equivalent cleanup is missing from {{obtainTokensAndScheduleRenewal}} /
{{{}updateTokensTask{}}}.
h4. Impact
Long-running applications that renew delegation tokens (keytab, or ccache-based
renewal) accumulate one UGI's worth of cached {{FileSystem}} handles per renewal
interval. With default settings the renewal interval is on the order of hours,
so the
growth is slow, but it is unbounded and can lead to gradually increasing memory
and
open file/connection usage (and eventually resource exhaustion) over the
lifetime of a
driver that runs for days or weeks.
The proxy-user branch of {{doLogin()}} returns the shared {{getCurrentUser()}}
and must
NOT be closed; the guard {{!currentUser.equals(freshUGI)}} correctly excludes
it.
h4. Suggested fix
Mirror the one-shot path: after the renewal {{doAs}} completes, close the fresh
UGI's
cached filesystems, guarded by {{!currentUser.equals(freshUGI)}} so the shared
current/proxy user is never closed.
Notes
Discovered during review of SPARK-38954 (PR #57285); it is a pre-existing issue
in the
Kerberos renewal path and is being tracked separately per reviewer request so
that
SPARK-38954 keeps the Kerberos paths unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
