[
https://issues.apache.org/jira/browse/IGNITE-24556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17936490#comment-17936490
]
Mikhail Efremov edited comment on IGNITE-24556 at 3/18/25 2:22 PM:
-------------------------------------------------------------------
During the colocation track (IGNITE-22115) a question about in-memory feature
intergration was emerged. So we have to figure out how to do this and what
technical decisions should be done about creation of 3 types of storages:
# table storages that are storing user data and snapshots;
# RAFT storages that are storing RAFT-related information like RAFT-log;
# transactions states storages.
At the moment before the colocation there was created only one table-partition
replication group with all 3 types of storages per table. After the colocation
we should have one zone-partition replication group with one RAFT node and its
storage per zone, one transactions states storage per zone and N table storages
per table in the zone.
Also we should consider that transactions states storage is presented as
*persistent only*. This solution was done because a transaction may works
across both volatile and persistent table set. Thus, before the colocation
{{isVolatile}} flag belongs to table and the whole its replication group, but
after IGNITE-22115 we should have a rule how to enable or disable the flag per
zone with mix of persistent and in-memory storages.
Mostly, the question is related to RAFT storages: tables' ones depend on
storage profile from table creation statement. Certain storage profile triggers
certain storage engine to create a certain instance of table storage. The
{{isVolatile}} flag affects only on RAFT storage.
Previously, {{isVolatile}} flag was taken from internal table implementation
and is passed to RAFT group starting method. So we could start set of storages
with in-memory per table individually. During colocation we should find a rule
that maintains that previous behavior for in-memory tables or/and partially
breaks compatibility. Besides we should consider the follow current in-memory
implementation issues:
# one out of three state (transactions states storage) is persistent only that
degrades exected performance benefit of in-memory mode that looks
counterintuitive;
# as before the colocation as after after a node restart we may lose user data
if a snapshot of a volatile table storeage and volatile RAFT log comaction were
made before, because the snapshot is used for recovery and in volatile case we
doesn't save the snapshot on a disk.
The second issue can not be solved through a combination of persistent RAFT
storage with volatile table storage, moreover we may consider that in-memory
approach assumes data lost due to nodes or a cluster restarts. Nevetheless we
should take into the account that in-memory performance always will be
resticted from the top by persistent transactions states storages.
All in all we may have three possible solutions:
|| || all volitile || mix || all persistent ||
| TxState | P | P | P |
| Raft | V | ? | P |
| MvPartition | V | ? | P |
# For a zone with any persistent storage we consider {{isVolatile}} as
{{false}} and iif all zone's storages are volatile then {{isVolatile}} is
{{true}}. Only in the last case we create both volatile RAFT and table
storages. Otherwise we create persistent RAFT storage and a table storage
depends on its storage profile.
#* Pros:
#** Clear approach that preserves previous in-memory logic as expected from
user's point of view (volatile storages both table and RAFT are created only
with explicit usage of in-memory mode.
#** Top possible performance (restricted by transactions states storages).
#* Cons:
#** Demands a new restrictions on a zone alterations that deny an alteration
that is trying to add a persistent profile to a zone that originally was
created with volatile-only storage profiles.
#** User data may be lost, but this behavior may be consdered as expected.
# For a zone with any volatile storage we consider {{isVolatile}} as {{true}}
and will start volatile RAFT storage in this case
#* Cons:
#** User may loses data with expected persistent behavior for petsistent
profile tables in such volatile zone.
#** Performance is still restricted by transactions states storage.
# Always consider {{isVolatile}} as {{false}} until all issues related to
in-memory track will be fixed.
After discussions there was a chosen solution from p.1. This means that:
# IGNITE-22391 should be closed because there nothing to do, because replicas
will use:
#* Persistent only transactional state storage.
#* RAFT storage depends on {{isVolatile}} flag is discussed above.
#* Table storage in table processors that are created and loaded to a
zone-replica in {{TableManager}} depends on storage profile during a table
creation.
# IGNITE-24371 should be updated with the decision above.
# IGNITE-24828 is created for new restrictions on a zone alteration
implementation.
# IGNITE-24829 is created for a documentation for p.3 changes.
was (Author: JIRAUSER303791):
During the colocation track (IGNITE-22115) a question about in-memory feature
intergration was emerged. So we have to figure out how to do this and what
technical decisions should be done about creation of 3 types of storages:
# table storages that are storing user data and snapshots;
# RAFT storages that are storing RAFT-related information like RAFT-log;
# transactions states storages.
At the moment before the colocation there was created only one table-partition
replication group with all 3 types of storages per table. After the colocation
we should have one zone-partition replication group with one RAFT node and its
storage per zone, one transactions states storage per zone and N table storages
per table in the zone.
Also we should consider that transactions states storage is presented as
*persistent only*. This solution was done because a transaction may works
across both volatile and persistent table set. Thus, before the colocation
{{isVolatile}} flag belongs to table and the whole its replication group, but
after IGNITE-22115 we should have a rule how to enable or disable the flag per
zone with mix of persistent and in-memory storages.
Mostly, the question is related to RAFT storages: tables' ones depend on
storage profile from table creation statement. Certain storage profile triggers
certain storage engine to create a certain instance of table storage. The
{{isVolatile}} flag affects only on RAFT storage.
Previously, {{isVolatile}} flag was taken from internal table implementation
and is passed to RAFT group starting method. So we could start set of storages
with in-memory per table individually. During colocation we should find a rule
that maintains that previous behavior for in-memory tables or/and partially
breaks compatibility. Besides we should consider the follow current in-memory
implementation issues:
# one out of three state (transactions states storage) is persistent only that
degrades exected performance benefit of in-memory mode that looks
counterintuitive;
# as before the colocation as after after a node restart we may lose user data
if a snapshot of a volatile table storeage and volatile RAFT log comaction were
made before, because the snapshot is used for recovery and in volatile case we
doesn't save the snapshot on a disk.
The second issue can not be solved through a combination of persistent RAFT
storage with volatile table storage, moreover we may consider that in-memory
approach assumes data lost due to nodes or a cluster restarts. Nevetheless we
should take into the account that in-memory performance always will be
resticted from the top by persistent transactions states storages.
All in all we may have three possible solutions:
|| || all volitile || mix || all persistent ||
| TxState | P | P | P |
| Raft | V | ? | P |
| MvPartition | V | ? | P |
# For a zone with any persistent storage we consider {{isVolatile}} as
{{false}} and iif all zone's storages are volatile then {{isVolatile}} is
{{true}}. Only in the last case we create both volatile RAFT and table
storages. Otherwise we create persistent RAFT storage and a table storage
depends on its storage profile.
#* Pros:
#** Clear approach that preserves previous in-memory logic as expected from
user's point of view (volatile storages both table and RAFT are created only
with explicit usage of in-memory mode.
#** Top possible performance (restricted by transactions states storages).
#* Cons:
#** Demands a new restrictions on a zone alterations that deny an alteration
that is trying to add a persistent profile to a zone that originally was
created with volatile-only storage profiles.
#** User data may be lost, but this behavior may be consdered as expected.
# For a zone with any volatile storage we consider {{isVolatile}} as {{true}}
and will start volatile RAFT storage in this case
#* Cons:
#** User may loses data with expected persistent behavior for petsistent
profile tables in such volatile zone.
#** Performance is still restricted by transactions states storage.
# Always consider {{isVolatile}} as {{false}} until all issues related to
in-memory track will be fixed.
After discussions there was a chosen solution from p.1. This means that:
# IGNITE-22391 should be closed because there nothing to do, because replicas
will use:
#* Persistent only transactional state storage.
#* RAFT storage depends on {{isVolatile}} flag is discussed above.
#* Table storage in table processors that are created and loaded to a
zone-replica in {{TableManager}} depends on storage profile during a table
creation.
# IGNITE-24371 should be updated with the decision above.
# IGNITE-24828 is created for new restrictions on a zone alteration
implementation.
# IGNITE-24829 is created for a documentation for p.3 changes.
> Refine in-memory storage support within colocation track
> --------------------------------------------------------
>
> Key: IGNITE-24556
> URL: https://issues.apache.org/jira/browse/IGNITE-24556
> Project: Ignite
> Issue Type: Task
> Reporter: Alexander Lapin
> Assignee: Mikhail Efremov
> Priority: Major
> Labels: ignite-3
>
> h3. Motivation
> We have two ticket for in-memory storage support
> * https://issues.apache.org/jira/browse/IGNITE-22391
> * https://issues.apache.org/jira/browse/IGNITE-24371
> First one lacks description. Let's refine what is expected to be done within
> aforementioned tickets and estimate them.
> h3. Definition of Done
> * Meaningful description is added to IGNITE-22391
> * It's clear how to implement given tickets.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
