[
https://issues.apache.org/jira/browse/HBASE-28044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782239#comment-17782239
]
Wellington Chevreuil commented on HBASE-28044:
----------------------------------------------
Yes, still using local disk. BucketCache originally already persists the map in
a local disk. In HBASE-27686/HBASE-28004 we are doing it periodically, to make
sure we can survive RS crashes, however the longer the period for saving the
map, less blocks would be recovered. This proposal does the saving
incrementally. Maybe we can work on an alternative storage solution for the
cache map on a separate jira?
> Reduce frequency of saving backing map in persistence cache
> -----------------------------------------------------------
>
> Key: HBASE-28044
> URL: https://issues.apache.org/jira/browse/HBASE-28044
> Project: HBase
> Issue Type: Sub-task
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Major
>
> Currently we always write the whole cache mapping into the persistent map
> file. This is not a lightweight operation, on a full 1.6TB cache with ten
> millions of block, this can grow as much as 10GB. In the current persistent
> cache implementation, we flush it to disk every 1s. If we raise the
> "checkpoint" period, we risk lose more cache events in the event of a
> recovery.
> This proposes reduce the frequency needed to save the backing map as follows:
> 1) Save every block addition/eviction into a single file in disk;
> 2) Checkpoint at higher intervals, consolidating all transactions into the
> larger map file;
> 3) In the event of failure, recovery would consist of loading the latest map
> file, then applying all the transactions files sequentially;
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
