[
https://issues.apache.org/jira/browse/HDFS-16518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei Yang updated HDFS-16518:
----------------------------
Description:
KeyProvider implements Closable interface but some custom impl of KeyProvider
needs explicit close in KeyProviderCache.
KeyProvider gets closed in KeyProviderCache only when cache entry is expired or
removed. In some cases, this is not happening(Some non-daemon threads in
KeyProvider implementation prevent jvm from shutting down).
This patch is to allow jvm shutdownhook to explicitly cleanup cache entries and
close KeyProvider immediately after filesystem instance gets closed.
{code:java}
Class KeyProviderCache
...
public KeyProviderCache(long expiryMs) {
cache = CacheBuilder.newBuilder()
.expireAfterAccess(expiryMs, TimeUnit.MILLISECONDS)
.removalListener(new RemovalListener<URI, KeyProvider>() {
@Override
public void onRemoval(
@Nonnull RemovalNotification<URI, KeyProvider> notification) {
try {
assert notification.getValue() != null;
notification.getValue().close();
} catch (Throwable e) {
LOG.error(
"Error closing KeyProvider with uri ["
+ notification.getKey() + "]", e);
}
}
})
.build();
}{code}
was:
We need to make sure the underlying KeyProvider used by multiple DFSClient
instances is closed at one shot during jvm shutdown. Within the shutdownhook,
we invalidate the cache and make sure they are all closed. The cache has a
removeListener hook which is called when cache entry is invalidated.
{code:java}
Class KeyProviderCache
...
public KeyProviderCache(long expiryMs) {
cache = CacheBuilder.newBuilder()
.expireAfterAccess(expiryMs, TimeUnit.MILLISECONDS)
.removalListener(new RemovalListener<URI, KeyProvider>() {
@Override
public void onRemoval(
@Nonnull RemovalNotification<URI, KeyProvider> notification) {
try {
assert notification.getValue() != null;
notification.getValue().close();
} catch (Throwable e) {
LOG.error(
"Error closing KeyProvider with uri ["
+ notification.getKey() + "]", e);
}
}
})
.build();
}{code}
> Cached KeyProvider in KeyProviderCache should be closed with
> ShutdownHookManager
> --------------------------------------------------------------------------------
>
> Key: HDFS-16518
> URL: https://issues.apache.org/jira/browse/HDFS-16518
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 2.10.0
> Reporter: Lei Yang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> KeyProvider implements Closable interface but some custom impl of KeyProvider
> needs explicit close in KeyProviderCache.
> KeyProvider gets closed in KeyProviderCache only when cache entry is expired
> or removed. In some cases, this is not happening(Some non-daemon threads in
> KeyProvider implementation prevent jvm from shutting down).
> This patch is to allow jvm shutdownhook to explicitly cleanup cache entries
> and close KeyProvider immediately after filesystem instance gets closed.
> {code:java}
> Class KeyProviderCache
> ...
> public KeyProviderCache(long expiryMs) {
> cache = CacheBuilder.newBuilder()
> .expireAfterAccess(expiryMs, TimeUnit.MILLISECONDS)
> .removalListener(new RemovalListener<URI, KeyProvider>() {
> @Override
> public void onRemoval(
> @Nonnull RemovalNotification<URI, KeyProvider> notification) {
> try {
> assert notification.getValue() != null;
> notification.getValue().close();
> } catch (Throwable e) {
> LOG.error(
> "Error closing KeyProvider with uri ["
> + notification.getKey() + "]", e);
> }
> }
> })
> .build();
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
