[
https://issues.apache.org/jira/browse/HBASE-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893148#action_12893148
]
Jonathan Gray commented on HBASE-2663:
--------------------------------------
The implementation of the lru was originally intended to put all overhead at
eviction time and as little as possible for reads/writes. This included an
attempted lock-free design based around the ConcurrentHashMap as the data
structure for blocks.
Given we're in java land, this background memory usage still impacts us, not
sure how significant, but for sure it's lots of objects.
You could get rid of the shallow copy into the structure that sorts by
accessTime by keeping an up-to-date linked list for each priority. I think you
need to roll your own LL implementation because the java one won't let you get
the LL pointers referenced into some other object (you want a LL but you also
need a constant time operation like removeAndPushToFront). Needs to be
thread-safe too, blocks being updated by concurrent reads and a concurrent
eviction process.
You'd still have to iterate the entire list and calculate the total sizes for
each priority and you'll still need the hash map, right?
So then, in the end, are you using the same amount of memory but keeping it
around and maintaining it in order versus re-allocating and calculating on the
fly at eviction time?
> LRU cache makes needless datastructure copies during eviction
> -------------------------------------------------------------
>
> Key: HBASE-2663
> URL: https://issues.apache.org/jira/browse/HBASE-2663
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Todd Lipcon
>
> Was browsing the LRU eviction code and came upon some very inefficient code.
> When we do eviction, BlockBucket.free() calls queue.get() which first inserts
> everything from the PriorityQueue<Block> into a LinkedList, then copies that
> entire linked list into an array. We then iterate over usually just a small
> percentage of the array to free some blocks until we have freed the requested
> amount.
> We ought to be able to just pull items out of the PriorityQueue directly and
> avoid all the churn.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
