[
https://issues.apache.org/jira/browse/HDFS-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16312580#comment-16312580
]
Yiqun Lin edited comment on HDFS-12802 at 1/5/18 9:11 AM:
----------------------------------------------------------
[~elgoiri], patch looks good to me, only two comments:
* It would be better to define configs {{MAX_CACHE_TIME_MILLIS_KEY}} and
{{MAX_CACHE_TIME_MILLIS_DEFAULT}} in {{DFSConfigKeys}} rather than in class
{{MountTableResolver}}. It Then document this in {{hdfs-default.xml}}.
* In test {{testCacheCleaning}}, we can use {{TEST_MAX_CACHE_SIZE}} to replace
{{10}} here {{assertTrue(cacheSize <= 10)}}.
I have one another thought but not must need to change in this JIRA: Can we use
the util class {{com.google.common.cache.Cache}} in guava for control this? It
will simplify the code a bunch I think. The maximum number size and expire time
of cache can be set in guava Cache.
Sample codes
{code}
Cache<String, String> cache = CacheBuilder.newBuilder()
.expireAfterAccess(2000, TimeUnit.MILLISECONDS)
.maximumSize(10)
.removalListener(new RemovalListener<String, String>() {
@Override
public void onRemoval(RemovalNotification<String, String>
removalNotification) {
System.out.println("Removing " +
removalNotification.getKey());
}
}).build();
cache.put("1", "2");
cache.put("2", "3");
System.out.println("Sleeping");
Thread.sleep(3000);
System.out.println("Re-put");
// latest access behavior to cache will trigger clean-up operation
cache.put("3", "4");
System.out.println("Size: " + cache.size());
{code}
was (Author: linyiqun):
[~elgoiri], I have one another thought: Can we use the util class
{{com.google.common.cache.Cache}} in guava for control this? It will simplify
the code a bunch I think. The maximum number size and expire time of cache can
be set in guava Cache.
Sample codes
{code}
Cache<String, String> cache = CacheBuilder.newBuilder()
.expireAfterAccess(2000, TimeUnit.MILLISECONDS)
.maximumSize(10)
.removalListener(new RemovalListener<String, String>() {
@Override
public void onRemoval(RemovalNotification<String, String>
removalNotification) {
System.out.println("Removing " +
removalNotification.getKey());
}
}).build();
cache.put("1", "2");
cache.put("2", "3");
System.out.println("Sleeping");
Thread.sleep(3000);
System.out.println("Re-put");
// latest access behavior to cache will trigger clean-up operation
cache.put("3", "4");
System.out.println("Size: " + cache.size());
{code}
> RBF: Control MountTableResolver cache size
> ------------------------------------------
>
> Key: HDFS-12802
> URL: https://issues.apache.org/jira/browse/HDFS-12802
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Íñigo Goiri
> Assignee: Íñigo Goiri
> Attachments: HDFS-12802.000.patch, HDFS-12802.001.patch,
> HDFS-12802.002.patch
>
>
> Currently, the {{MountTableResolver}} caches the resolutions for the
> {{PathLocation}}. However, this cache can grow with no limits if there are a
> lot of unique paths. Some of these cached resolutions might not be used at
> all.
> The {{MountTableResolver}} should clean the {{locationCache}} periodically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
