[
https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671535#action_12671535
]
Jonathan Gray commented on HBASE-1186:
--------------------------------------
I didn't have time to finish work on this yesterday. I just posted up what I
had finished before I left, it's definitely not done yet.
Will properly format it, etc well before it's ready to be included.
Re: 32-bit vs 64-bit there are some tests that can be used. My current plan is
to go forward with assuming 64bit and in the future we can easily add something
to detect like you said. It would then just change the size of REFERENCE which
would bubble up into anything implementing HeapSize which should be using that
static to compute its size.
There is some stuff left over from other code in there (transient datamembers
and such) that I will be taking out. I see no need for serialization. Same
goes for making things private, have been working on the lru and heapsize
elements and not so much the rest of the code. Will clean it up next week.
Currently, maxMemUsage is passed in. This LRU is intended for cell cache which
I think should be a per-regionserver setting. This can be adapted to work for
the block cache as well in any number of ways. There are a few reasons I don't
think softrefs are the way to go for block caching, definitely not the way to
go for cell caching. Next week I'll put together the proposal for why I think
we should use our own LRU mechanism rather than relying on softrefs across the
board. The primary issues are how this will interact with existing
memcache/heap monitoring, unpredictability of softref eviction, and the ability
to implement in-memory tables very simply if we have our own lru mechanism
(implemented in the way described in the bigtable paper).
Same goes for synchronization. This is an early version not ready for prime
time. Not sure what my plan is yet will post thoughts here. Will be working
on this early in the week.
Thanks for the review stack.
> Memory-aware Maps with LRU eviction
> -----------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: HeapSize.java, LruHashMap.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to
> manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache.
> Erik Holstad has done extensive testing and benchmarking and will post
> results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in
> HBASE-61. It should try to use all available memory, but must contend with
> Memcaches so is resizable to deal with heap pressure. Adding high priority
> blocks (evicted last) gives us in-memory functionality as described in
> bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached
> objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
