[
https://issues.apache.org/jira/browse/RATIS-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097760#comment-18097760
]
Ivan Andika edited comment on RATIS-2597 at 7/21/26 7:14 AM:
-------------------------------------------------------------
Additionally, in SCM we have a distributed sequence ID generator using Ratis.
However, since the Raft log and StateMachineUpdater is shared unrelated Raft
group for allocating containers which increases the write latency, see
HDDS-15634.
was (Author: JIRAUSER298977):
For example, in SCM we have a distributed sequence ID generator on Ratis.
However, since the Raft log and StateMachineUpdater is shared unrelated Raft
group for allocating containers which increases the write latency, see
HDDS-15634.
> Local StateMachine Sharding within a Ratis group
> ------------------------------------------------
>
> Key: RATIS-2597
> URL: https://issues.apache.org/jira/browse/RATIS-2597
> Project: Ratis
> Issue Type: New Feature
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Simply an idea.
> Currently, each Ratis group contains a single Raft log and StateMachine (+
> updater). This makes every Ratis transactions to be serialized. However, in
> the case of key value store there are possible cases where there are some
> independent shard (e.g. prefix) where operations is guaranteed to be within
> that shard). In this case, the single StateMachine can cause operations
> across independent shards to content with each other. In the context of
> Apache Ozone, currently metadata operations are independent for different
> buckets, but since Ozone uses a single Ratis group for the namespace, one
> bucket operations can block other bucket operations.
> We can technically create multiple Ratis groups where each Ratis group is a
> shard. This is usually done by either creating a static sharding mechanism
> (e.g. divide the namespace to 256 shards) or using a separate shard manager
> that will be able to do split, merge a single namespace to multiple shards
> (e.g. TiKV, HBase regions, CockroachDB, etc). However, usually this causes
> more complexity (especially the later), since we have to manage N Raft
> groups, each with their own leader and follower combination. Additionally, in
> the case of Ozone, we want OM to still share a single RocksDB state so that
> we can halt the Raft group if there is a InstallSnapshot. This is very
> difficult if we have multiple independent Raft groups.
> What if we can make a single Raft group be transparently contain multiple
> Raft logs, each with a single StateMachineUpdater. The Raft group contains
> only a single leader and followers combination (unlike multiple Raft groups),
> but underneath there are sharded logs that can be used to improve the
> paralellism. This way, the Raft group is transparent to client and client
> does not need to keep track of different Raft groups with different leaders.
> One requirement is that each request now needs to specify a shard key (e.g.
> in Ozone case the shard key is the bucket name or ID) so that it can be
> routed to the correct StateMachine shard. In the future, we can have some
> kind of 2PC protocol if we want support cross-shard transactions.
> Drawbacks
> * The leader for sharded Ratis is shared among the shards which can be the
> new bottleneck
> * Raft implementation can become more complicated since we need to keep
> track of the states of each shard.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
