GitHub user Abhinav2656 created a discussion: [Proposal] Multi-Tenant Namespace Isolation in Cluster Mode(issue #3231)
Hi @jihuayu and @PragmaTwice, Thanks again for the discussion. Based on your feedback, I’ve put together an architectural blueprint for properly supporting namespaces in cluster mode. It addresses both the migration iterators and the slot mapping isolation. I'd love to get your thoughts on this before I start writing code. 1. The Migration & Cleanup Iterators The Issue: Right now, SlotMigrator::sendSnapshotByCmd() and sendSnapshotByRawKV() initialize their RocksDB iterators using the redis::Database default namespace_ (__namespace). Because of this, the migration loop completely ignores all tenant-prefixed keys within the target slot. The Fix: We need to decouple slot migration from a single namespace. -Namespace Registry: We can introduce a lightweight namespace cache/registry (likely in Server or Storage) that maintains a list of all active namespaces retrieved from the metadata. -Multi-Pass Iterator: Inside slot_migrate.cc and cluster.cc (for ClearKeysOfSlotRange), we refactor the logic to loop over this namespace registry. -For each active namespace ns, the migrator dynamically constructs the prefix: ComposeSlotKeyPrefix(ns, slot_range.start). This ensures that when a slot moves, all keys mapping to that slot across all namespaces are migrated transactionally before the setForbiddenSlotRange barrier is lifted. 2. Slot Topology Isolation The Issue: Currently, cluster.cc utilizes a single, global std::shared_ptr<ClusterNode> slots_nodes_[kClusterSlots] array to route traffic. If multiple tenants operate on the same physical nodes, a heavy operation by Tenant A on Slot 100 will directly degrade Tenant B’s performance on that same slot. The Fix: We have two potential paths for this, and I want your consensus on which fits the KVrocks philosophy best: Option 1: Global Topology (Shared routing)We maintain the single slots_nodes_ array. All namespaces share the exact same cluster topology. If Slot 100 maps to Node A, it maps to Node A for everyone. Pros: Minimal changes required in cluster.cc. Routing logic remains fast ($O(1)$). Cons: Lacks strict physical tenant isolation. Resource spikes can bleed across namespaces. Option 2: Isolated Topology (Strict Multi-Tenancy)We refactor the routing table into a map: std::unordered_map<std::string, std::array<std::shared_ptr<ClusterNode>, kClusterSlots>> namespace_slots_nodes_. Pros: Absolute isolation. Tenant A can migrate its Slot 100 to Node B, while Tenant B keeps its Slot 100 on Node A. Cons: Requires a massive refactoring of the cluster.cc state machine, migration logic, and the CLUSTER NODES command. My Recommendation: Given the complexity of RocksDB compaction and the massive overhead of maintaining independent replication states per namespace, I recommend starting with Option 1 (Global Topology). It safely unblocks namespace usage in cluster mode via the iterator fix. We can always evaluate Option 2 in a subsequent phase if users demand strict physical isolation. Let me know if this aligns with your vision, particularly regarding the Global vs. Isolated topology decision. Once we have a consensus, I'll dive into the implementation! GitHub link: https://github.com/apache/kvrocks/discussions/3508 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
