[
https://issues.apache.org/jira/browse/HDFS-8892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14710959#comment-14710959
]
Ravikumar commented on HDFS-8892:
---------------------------------
I am fine if this runs in a separate thread as you said…
My app opens hadoop files only once. The block input-streams, once opened need
to stay that way forever until app instructs to close it. The close call
happens only for deleting the file or transferring the ownership of file to
another server {Like Hbase regions}
With current ShortCircuitCache, I am not able to find any knobs to tune for
such kind of behaviour.
I am planning to set all params of LRU cache high so that ShortCircuitCache
never closes any open input-stream by itself. All the stream-close calls are
handled only through invalid-slot-checker
> ShortCircuitCache.CacheCleaner can add Slot.isInvalid() check too
> -----------------------------------------------------------------
>
> Key: HDFS-8892
> URL: https://issues.apache.org/jira/browse/HDFS-8892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Affects Versions: 2.7.1
> Reporter: Ravikumar
> Assignee: kanaka kumar avvaru
> Priority: Minor
>
> Currently CacheCleaner thread checks only for cache-expiry times. It would be
> nice if it handles an invalid-slot too in an extra-pass of evictable map…
> for(ShortCircuitReplica replica:evictable.values()) {
> if(!scr.getSlot().isValid()) {
> purge(replica);
> }
> }
> //Existing code...
> int numDemoted = demoteOldEvictableMmaped(curMs);
> int numPurged = 0;
> Long evictionTimeNs = Long.valueOf(0);
> ….
> …..
> Apps like HBase can tweak the expiry/staleness/cache-size params in
> DFS-Client, so that ShortCircuitReplica will never be closed except when Slot
> is declared invalid.
> I assume slot-invalidation will happen during block-invalidation/deletes
> {Primarily triggered by compaction/shard-takeover etc..}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
