[
https://issues.apache.org/jira/browse/HBASE-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783441#comment-13783441
]
Alex Feinberg commented on HBASE-8894:
--------------------------------------
Vladimir,
Those are very legitimate issues:
1) One approach around the on-heap keys (not an issue in my setup as I was not
using file based cache, but certainly an issue with fusion IO) could be to use
a hash table (with extension array) or (in cases where the block index is
expected to not fit in ram) a b-tree over direct/memory-mapped byte buffers.
This would be tricky to implement but it has been done:
https://github.com/jankotek/MapDB/tree/master/src/main/java/org/mapdb
2) Eviction algorithm is indeed primitive (and also high on priority of things
to fix), but as far as I re-call eviction ( freeSpace() here --
https://github.com/apache/hbase/blob/0.89-fb/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java#L488-625
) only blocks draining to the ioEngine -- in other words, while the cache
space is being free you can still use read from the cache (this uses striped
locking) -- and writes will enter RAMCache and be queued for the ioEngine (
https://github.com/apache/hbase/blob/0.89-fb/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java#L648-660
). All IOEngine draining threads will be blocked during eviction, however --
that may be more problematic for file-based caches -- long draining may cause a
lot of entries to be built up in RAMCache. If the queue is full the threads
will be blocked, but you can configure to wait up to a maximum amount -- but
this doesn't affect actual writes to HBase, as with L2Cache writes only happen
during flushes (i.e., flushes will take longer if they happen during eviction).
Thanks,
- af
> Forward port compressed l2 cache from 0.89fb
> --------------------------------------------
>
> Key: HBASE-8894
> URL: https://issues.apache.org/jira/browse/HBASE-8894
> Project: HBase
> Issue Type: New Feature
> Reporter: stack
> Assignee: Liang Xie
> Priority: Critical
> Fix For: 0.98.0
>
>
> Forward port Alex's improvement on hbase-7407 from 0.89-fb branch:
> {code}
> 1 r1492797 | liyin | 2013-06-13 11:18:20 -0700 (Thu, 13 Jun 2013) | 43 lines
> 2
> 3 [master] Implements a secondary compressed cache (L2 cache)
> 4
> 5 Author: avf
> 6
> 7 Summary:
> 8 This revision implements compressed and encoded second-level cache with
> off-heap
> 9 (and optionally on-heap) storage and a bucket-allocator based on
> HBASE-7404.
> 10
> 11 BucketCache from HBASE-7404 is extensively modified to:
> 12
> 13 * Only handle byte arrays (i.e., no more serialization/deserialization
> within)
> 14 * Remove persistence support for the time being
> 15 * Keep an index of hfilename to blocks for efficient eviction on close
> 16
> 17 A new interface (L2Cache) is introduced in order to separate it from the
> current
> 18 implementation. The L2 cache is then integrated into the classes that
> handle
> 19 reading from and writing to HFiles to allow cache-on-write as well as
> 20 cache-on-read. Metrics for the L2 cache are integrated into
> RegionServerMetrics
> 21 much in the same fashion as metrics for the existing (L2) BlockCache.
> 22
> 23 Additionally, CacheConfig class is re-refactored to configure the L2
> cache,
> 24 replace multile constructors with a Builder, as well as replace static
> methods
> 25 for instantiating the caches with abstract factories (with singleton
> 26 implementations for both the existing LruBlockCache and the newly
> introduced
> 27 BucketCache based L2 cache)
> 28
> 29 Test Plan:
> 30 1) Additional unit tests
> 31 2) Stress test on a single devserver
> 32 3) Test on a single-node in shadow cluster
> 33 4) Test on a whole shadow cluster
> 34
> 35 Revert Plan:
> 36
> 37 Reviewers: liyintang, aaiyer, rshroff, manukranthk, adela
> 38
> 39 Reviewed By: liyintang
> 40
> 41 CC: gqchen, hbase-eng@
> 42
> 43 Differential Revision: https://phabricator.fb.com/D837264
> 44
> 45 Task ID: 2325295
> 7 ------------------------------------------------------------------------
> 6 r1492340 | liyin | 2013-06-12 11:36:03 -0700 (Wed, 12 Jun 2013) | 21 lines
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
