[
https://issues.apache.org/jira/browse/HDFS-16518?focusedWorklogId=746830&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746830
]
ASF GitHub Bot logged work on HDFS-16518:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Mar/22 19:41
Start Date: 23/Mar/22 19:41
Worklog Time Spent: 10m
Work Description: ibuenros commented on a change in pull request #4100:
URL: https://github.com/apache/hadoop/pull/4100#discussion_r833658315
##########
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java
##########
@@ -85,6 +90,26 @@ public KeyProvider call() throws Exception {
}
}
+ public static final int SHUTDOWN_HOOK_PRIORITY =
FileSystem.SHUTDOWN_HOOK_PRIORITY - 1;
+
+ private class KeyProviderCacheFinalizer implements Runnable {
+ @Override
+ public synchronized void run() {
+ invalidateCache();
+ }
+ }
+
+ /**
+ * Invalidate cache and auto close KeyProviders in the cache
+ */
+ @VisibleForTesting
+ synchronized void invalidateCache() {
+ LOG.debug("Invalidating all cached KeyProviders in ShutdownHookManager.");
Review comment:
This log is technically not correct in that we don't know the call is
coming form ShutdownHookManager. Maybe just remove the last two words from the
log.
##########
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java
##########
@@ -85,6 +90,26 @@ public KeyProvider call() throws Exception {
}
}
+ public static final int SHUTDOWN_HOOK_PRIORITY =
FileSystem.SHUTDOWN_HOOK_PRIORITY - 1;
+
+ private class KeyProviderCacheFinalizer implements Runnable {
+ @Override
+ public synchronized void run() {
+ invalidateCache();
+ }
+ }
+
+ /**
+ * Invalidate cache and auto close KeyProviders in the cache
+ */
+ @VisibleForTesting
+ synchronized void invalidateCache() {
+ LOG.debug("Invalidating all cached KeyProviders in ShutdownHookManager.");
+ if (cache != null) {
+ cache.invalidateAll();
Review comment:
Maybe add a comment that this will close the providers due to the cache
hook?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 746830)
Time Spent: 0.5h (was: 20m)
> Cached KeyProvider in KeyProviderCache should be closed with
> ShutdownHookManager
> --------------------------------------------------------------------------------
>
> Key: HDFS-16518
> URL: https://issues.apache.org/jira/browse/HDFS-16518
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 2.10.0
> Reporter: Lei Yang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We need to make sure the underlying KeyProvider used by multiple DFSClient
> instances is closed at one shot during jvm shutdown. Within the shutdownhook,
> we invalidate the cache and make sure they are all closed. The cache has a
> removeListener hook which is called when cache entry is invalidated.
> {code:java}
> Class KeyProviderCache
> ...
> public KeyProviderCache(long expiryMs) {
> cache = CacheBuilder.newBuilder()
> .expireAfterAccess(expiryMs, TimeUnit.MILLISECONDS)
> .removalListener(new RemovalListener<URI, KeyProvider>() {
> @Override
> public void onRemoval(
> @Nonnull RemovalNotification<URI, KeyProvider> notification) {
> try {
> assert notification.getValue() != null;
> notification.getValue().close();
> } catch (Throwable e) {
> LOG.error(
> "Error closing KeyProvider with uri ["
> + notification.getKey() + "]", e);
> }
> }
> })
> .build();
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
