[
https://issues.apache.org/jira/browse/IGNITE-15351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425119#comment-17425119
]
Andrey N. Gura commented on IGNITE-15351:
-----------------------------------------
[~ibessonov] I've took a look at the PR and have some comments. Could you
please take a look? Thanks.
> Research possibility of having caching layer on top of RocksDB partitions
> -------------------------------------------------------------------------
>
> Key: IGNITE-15351
> URL: https://issues.apache.org/jira/browse/IGNITE-15351
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 3.0.0-alpha2
> Reporter: Ivan Bessonov
> Assignee: Ivan Bessonov
> Priority: Major
> Labels: iep-74, ignite-3
> Fix For: 3.0.0-alpha3
>
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> In Ignite 2.x there's a concept of "Data Regions", which is basically a set
> of fixed-sized in-memory caches that store data for a number of cache groups
> (let's ignore system region and similar stuff for now). It is very convenient
> and represents a core design feature in Apache Ignite - In-Memory Database.
> Currently, Page Memory subsystem is not yet ported to Ignite 3.x codebase.
> Instead, there's an implementation based on RocksDB database to store data
> persistently.
> But, this implementation is very simple and naive. There's no notion of
> in-memory cache across multiple tables, meaning that it can't be called an
> In-Memory Database. We should investigate ways to add this concept back on
> top of RocksDB implementation.
> There are several things to investigate here:
> * how do you set up rocksdb properly and control its memory consumption - we
> should allow some configuration and a meaningful set of defaults;
> * how do you put a cache on top of several rocksdb instances. This is
> actually pretty easy, just use
> "org.rocksdb.Options#setRowCache(org.rocksdb.Cache)", it has LRU and Clock
> implementations. A way to configure it is still required;
> * how do we introduce data regions into our system? I see something like
> this:
> ** list of regions is either a node or cluster configuration;
> ** name of the region is a property of every individual table or table group
> (or whatever else we'll be having).
> Last proposition is a bit tricky, cause it won't look like "create table with
> rocks engine with Clock cache...", it would look like "create table in region
> Foo". We have to conceptualize all these things and come up with proper
> naming at least.
> h3. Update 1
> * the only way to control rocksdb memory usage is to have a single DB
> instance. For every table there will be several column families:
> ** one for table meta;
> ** one for every partition;
> ** one for every index;
> * data regions are a configuration of every individual node. They will have
> name, type and some other settings. The way tables chose the region remains
> to be defined;
> * there have to be common rocksdb settings outside of region settings, like
> mem table size, wal settings, etc.
> h3. Update 2
> * actually, there is a way to have a shared memory manager for several
> instances
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
