[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113492#comment-16113492 ]
Chen Liang commented on HDFS-12196: ----------------------------------- Thanks [~cheersyang] for the reply and the updated patch! I'm still concerned about the result caching though...I agree with you that it can save RPC calls, but the thing is it can be very tricky to get the caching work properly...more specifically: 1. One potential issue is that it seems there is no purging of the cache. i.e. once an entry gets added to the {{resultList}}, it never gets removed. So the size of {{resultList}} will monolithically increase to the point where no more entries can be added and then the result cache will no longer be helpful for any further entry. Also since all entries are just being added when there is space, if there is no more query for those cached result, without purging, we may end up just holding a bunch of objects in memory that will never be useful. I think ideally the cache should probably be removing entries based on factors such as access time, frequency etc. In short, the cache needs to have purging. 2. Seems there is no easy way to check the current {{resultList}} efficiently. Any service using it will have to check the entire cache. Namely, since it is a list, so even if some service wants to check if the result of a call is cached in it before making the call, it still needs to iterate over all the entries in it, so if the list is large enough, it could be even less efficient than contacting datanodes. In short, the cache needs to have fast lookup. Another side thing is that the entire {{resultList}} object is exposed and returned to caller, a bug in caller can easily mess up the list... I'm not against adding cache here, just saying we should probably pay a little attention. If we really want cache here, it shouldn't be hard to implement a cache on top of either [LRUMap|https://commons.apache.org/proper/commons-collections/apidocs/org/apache/commons/collections4/map/LRUMap.html] OR [Guava loading cache|https://github.com/google/guava/wiki/CachesExplained], which should make purging and fast lookup very easy, even trivial. (One example is the cache in {{XceiverClientManager}}, just FYI). > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > ---------------------------------------------------------------------------------------------- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone > Reporter: Weiwei Yang > Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org