GitHub user Aetherance edited a discussion: Proposal: Keyspace Notifications
for Kvrocks
## 1. Background and goals
Applications reading from a read-only Kvrocks replica keep a local cache to cut
latency and reduce round-trips to the primary. Keeping that cache fresh
requires a notification when a key changes. Kvrocks has no such mechanism
today, which blocks applications that depend on Redis Keyspace Notifications
from migrating.
This design implements the minimal Redis-compatible Keyspace Notifications
surface needed by #2915:
- `notify-keyspace-events` configuration.
- Events published to `__keyspace@<db>__:<key>` with payload = event name.
- Events published to `__keyevent@<db>__:<event>` with payload = key.
- Consumption through the existing `SUBSCRIBE` / `PSUBSCRIBE`.
- `SET` and `DEL` notifications on both primary and replica.
The MVP is intentionally narrow. It does not infer Redis command semantics from
RocksDB storage mutations alone: those records cannot distinguish `SET` from
`APPEND`, `SETRANGE`, or `INCR`, nor a genuine `DEL` command from lazy expiry
or internal delete-and-recreate flows. Instead, it reuses the existing
Redis-type LogData carried in the same write batch to distinguish the supported
`set` and `del` cases.
## 2. Architecture
Redis emits keyspace events at the command execution point, and replicas
re-emit them by replaying the replicated command stream. Kvrocks replicates the
RocksDB WAL (`WriteBatch`), not Redis commands, so replicas need explicit
command context when a storage mutation is ambiguous.
The design therefore extends the existing `redis::WriteBatchLogData` context in
the same `WriteBatch` as the data mutation:
```mermaid
flowchart TD
subgraph primary["Primary / standalone"]
cmd["Command layer<br/>SET / DEL succeeds"]
logdata["Create existing LogData<br/>for replicas"]
local_event["Keep command-derived<br/>notify fact"]
write["db_->Write(batch) succeeds"]
publish_primary["After write succeeds<br/>publish local fact"]
notify_primary["NotifyKeyspaceEvent"]
end
subgraph replica["Replica"]
recv["Receive replicated WriteBatch"]
apply["Apply batch succeeds"]
decode_replica["Extract notify events<br/>during batch iteration"]
notify_replica["NotifyKeyspaceEvent"]
end
pub["PublishMessage<br/>keyspace / keyevent"]
clients["Local SUBSCRIBE / PSUBSCRIBE subscribers"]
cmd --> logdata --> write --> publish_primary --> notify_primary
cmd --> local_event --> publish_primary
logdata -. "replicated with WriteBatch" .-> recv
recv --> apply --> decode_replica --> notify_replica
notify_primary --> pub
notify_replica --> pub
pub --> clients
```
This keeps the WAL as the replication carrier without guessing event names from
column-family writes. The primary already has command execution context, so it
keeps the local notification fact and publishes it only after `db_->Write`
succeeds. The replica publishes after it applies the same batch and
reconstructs the notification from the replicated LogData plus metadata
records. With the same local `notify-keyspace-events` setting, both roles
publish the same event names and payloads.
Pub/Sub fan-out is reused unchanged: `Server::PublishMessage` already handles
exact-channel and pattern subscriptions and is a pure in-memory operation, so
it works on read-only replicas.
## 3. Design decisions
### 3.1 Config: `notify-keyspace-events`
Add the `notify-keyspace-events` config. The default is `""` (disabled). The
MVP parser accepts the Kvrocks-supported subset of Redis flags `K E A g $ l s h
z t`, rejects unsupported or unknown characters, and supports runtime updates
through `CONFIG SET`.
Only `g` (`del`) and `$` (`set`) currently produce notifications. The remaining
data-type flags are accepted because Kvrocks has those storage types, but they
have no MVP emitter yet. Internally these flags are parsed into a new
notification bitmask, separate from the existing storage `RedisType` enum:
| Bit | Flag | Meaning |
|---:|---|---|
| `1 << 0` | `K` | publish keyspace channels |
| `1 << 1` | `E` | publish keyevent channels |
| `1 << 2` | `g` | generic events; MVP emits `del` |
| `1 << 3` | `$` | string events; MVP emits `set` |
| `1 << 4` | `l` | accepted, no MVP emitter |
| `1 << 5` | `s` | accepted, no MVP emitter |
| `1 << 6` | `h` | accepted, no MVP emitter |
| `1 << 7` | `z` | accepted, no MVP emitter |
| `1 << 8` | `t` | accepted, no MVP emitter |
In this MVP, `A` expands only to the accepted Kvrocks data-class subset
`g$lshzt`. Redis-only or out-of-scope flags such as `a`, `d`, `x`, `e`, `m`,
`n`, `o`, and `c` are intentionally not accepted until their event semantics
are implemented. Only `$` and `g` produce events, so the common `KEA` setting
enables both channel forms and lets `set` / `del` pass the filter.
The other classes enabled by `A` are accepted but inert for now. If later
releases add notification extraction for list/hash/etc., existing `KEA`
deployments will naturally start receiving those events; that matches Redis
semantics, but should be called out in release notes.
### 3.2 Reusing existing LogData
Kvrocks already writes `redis::WriteBatchLogData` into many write batches with
`WriteBatch::PutLogData()`. The current source model is:
```cpp
class WriteBatchLogData {
RedisType type_;
std::vector<std::string> args_;
};
```
It is encoded as the numeric Redis type followed by optional arguments.
Existing batch iterators already separate server-level log data from Redis-type
log data before decoding. This MVP keeps that boundary and only extends
Redis-type `WriteBatchLogData`; it does not add a new top-level log-data kind.
The MVP reuses that mechanism and slightly extends the optional arguments with
notification command context, for example by adding Redis command codes such as
`kRedisCmdSet` and `kRedisCmdDel`. New command codes must be appended without
renumbering existing `RedisCommand` values. It does not add an independent
`kse1` record and does not put raw key names into LogData arguments. The
affected namespace and key are still decoded from the existing metadata
column-family `PutCF` / `DeleteCF` records with `ExtractNamespaceKey`, which
preserves raw key bytes and avoids adding one log record per key.
Primary-subkey and stream records use `InternalKey` elsewhere and are ignored
by the `del` extractor.
A new or extended replica-side notification extractor is stateful in the same
way as the existing `WriteBatchExtractor`: `LogData` establishes the current
Redis type and optional command context, and subsequent metadata-column-family
records can produce notifications when both the LogData context and the
column-family operation match a supported event.
- `set`: requires a Redis string LogData context explicitly marked as a
supported SET command path, then a metadata `PutCF` whose decoded metadata type
is `kRedisString`.
- `del`: requires a LogData context explicitly marked as the supported DEL
command path, then a metadata `DeleteCF`. It must not fire for primary-subkey
deletes such as `HDEL`, `SREM`, `ZREM`, or list element removals.
The LogData must precede the records it describes, matching current write-batch
patterns. If the context is missing, malformed, a server log, or unrelated to
key notifications, the extractor skips notification emission for those records.
Decode errors are non-fatal in the notification pass. A malformed or unknown
LogData command context is logged and skipped for notification purposes; it
must not fail an already committed write or block replication.
LogData command context records semantic facts, not publication decisions. They
are written only when a command has definitely performed a supported mutation.
Each node still applies its own `notify-keyspace-events` filter when publishing.
LogData write policy:
- If the primary's `notify-keyspace-events` enables no supported class bit (`g`
/ `$`), supported commands may skip adding notification command context to
avoid default-off overhead. The skip keys only on class bits, never on the `K`
/ `E` channel selectors, so a primary with a class bit but no selector still
carries the context for replicas to publish.
- If the primary skips notification context, replicas cannot publish
notifications for that write even if their local config is later enabled (the
operational consequence for replica caches is stated as a guarantee in §5).
- When enabled, each supported batch adds only a small command-context
extension to existing Redis LogData; the affected keys come from the normal
write records already present in the WAL and replication stream.
Configuration is thus checked twice — at LogData-context write time on the
primary and at publish time on each node; a `CONFIG SET` that disables
notifications between the two still ships the context to replicas but
suppresses the primary's local publish.
### 3.3 Command paths
Only commands with unambiguous MVP semantics add notification command context:
- `SET`: mark the batch as a supported `set` producer only when the literal
`SET` command actually writes the key. Conditional variants such as `SET key
value NX` must not add notification context when the condition fails. Shared
string helpers must not infer `set` from `kRedisString` alone, because other
string commands also write string metadata.
- `DEL`: mark the batch as a supported `del` producer only when the literal
`DEL` command deletes at least one key. The extractor emits one `del` event per
metadata `DeleteCF`, each carrying that single key. Missing keys do not produce
write records or notifications. `UNLINK` currently shares `CommandDel` and
`MDel`, so the implementation must distinguish the command name or command
attributes and avoid enabling `UNLINK` unless it is explicitly added to the MVP
scope.
Every other write path (the out-of-scope list in §4) adds no notification
command context in this MVP. This is deliberate: emitting no event is safer
than emitting a Redis event name that does not match the command semantics.
### 3.4 Event source and trigger points
Primary and replica publish at different points, and only the replica
reconstructs notifications from LogData:
- The primary command path keeps a local notification fact for a successful
literal `SET` / `DEL`, then calls `Server::NotifyKeyspaceEvent` only after
`db_->Write` succeeds. It does not infer primary notifications from LogData or
column-family records.
- The replica applies the replicated batch first, then extracts notifications
from LogData and metadata records while iterating that same batch, reusing the
path already present around `parseWriteBatch`.
Supported command paths pass notification command context to the type/storage
operation that creates the existing `WriteBatchLogData` for replicas, but those
type/storage operations do not publish. On the primary, publication is a single
after-write step from the retained local fact. This keeps failed writes silent
and prevents a single-node write from publishing once from the command path and
once from the batch path.
Publication is bound only to live write/apply paths. During restart, RocksDB
replays WAL internally inside `DB::Open`; that recovery does not invoke the
notification path, so historical LogData is not republished.
Decode errors are non-fatal in the notification handler. Malformed or
unsupported notification context is logged and skipped; keyspace notifications
must never abort a committed write or stall replication.
### 3.5 Event emitter
`Server::NotifyKeyspaceEvent` applies the local config filter and reuses the
existing Pub/Sub path to publish the two Redis-compatible channel forms:
For a successful `SET foo bar` in the default namespace, `KEA` publishes:
```text
__keyspace@0__:foo -> "set"
__keyevent@0__:set -> "foo"
```
For a successful `DEL foo`, it publishes:
```text
__keyspace@0__:foo -> "del"
__keyevent@0__:del -> "foo"
```
### 3.6 `<db>` / namespace mapping
For Redis compatibility, the default namespace maps to db `0`, so standard
subscriptions such as `PSUBSCRIBE __keyevent@0__:*` work out of the box.
Non-default Kvrocks namespaces cannot be inserted into `<db>` directly. A
namespace literally named `"0"` would collide with Redis db `0` channels and
could leak events across tenants. The mapping must therefore be collision-free.
`MapNamespaceToKeyspaceDB(ns)` is defined as:
- default namespace -> `"0"`;
- non-default namespace -> `"ns:" + PercentEncode(ns)`, where percent encoding
escapes every byte outside `[A-Za-z0-9_.-]` and also escapes `%`.
Subscribers to `__keyevent@0__:*` therefore only see the default namespace.
Subscribers that opt into non-default namespaces use the encoded form, for
example `__keyevent@ns:tenantA__:*`. The payload remains only the key name, so
namespace identity comes from the channel.
The key component in `__keyspace@<db>__:<key>` and the payload in
`__keyevent@<db>__:<event>` are the raw user key bytes, matching Redis
behavior. Keys are not percent-encoded, even if they contain `:`, whitespace,
or binary bytes.
## 4. Event scope
Supported MVP events:
| Command | Event class | Event | Emitted when |
|---|---|---|---|
| `SET` | `$` | `set` | the command writes the key |
| `DEL` | `g` | `del` | the key is actually deleted |
Out of scope:
- `UNLINK`, `APPEND`, `SETRANGE`, `GETSET`, `INCR`, and other string commands.
- Hash/list/set/zset/stream element updates.
- `EXPIRE`, `PEXPIRE`, `expired`, and lazy/compaction expiry.
- `FLUSHDB` / `FLUSHALL` and `DeleteRangeCF` notifications.
- `rename_from`, `rename_to`, `new`, `keymiss`, `evicted`, and module events.
`FLUSHDB` is explicitly a limitation for cache invalidation: this MVP does not
emit per-key or aggregate flush notifications. Applications that rely on flush
visibility need a later design for aggregate `flushdb` / `flushall` events.
## 5. Semantics and guarantees
- Best effort only. Notifications are Pub/Sub messages: they are not persisted,
acknowledged, replayed, or retried.
- Failed writes publish nothing. LogData context in an uncommitted batch has no
visible effect because publication only runs after successful `db_->Write` or
replica apply.
- A write commits before its local notification publishes; a crash in that
window can make local subscribers miss the event. Replica notifications publish
only after apply, so they can lag the primary, and a replica crash between
apply and publish loses the event — restart recovery does not replay it, which
also prevents duplicate notifications after crashes.
- `notify-keyspace-events` is node-local, and the primary's setting gates
notification LogData context (§3.2). A primary with no supported class bit
writes no notification context, so its replicas publish nothing no matter how
they are configured. For the #2915 replica-cache use case operators must
therefore enable a supported class bit (for example `KEA`) on the primary —
even if it has no local subscribers — and set the same value on every
subscribed replica.
- Rolling upgrades rely on older nodes tolerating the slightly extended
`WriteBatchLogData` arguments. Older nodes may ignore the notification context
and not publish, but they must still apply and replicate the write batch.
- There is no cross-node de-duplication: a client subscribed to both primary
and replica may process the same logical mutation twice.
- For one affected key, keyspace is published before keyevent on one node. No
ordering is promised across different affected keys — multi-key commands like
`DEL k1 k2 k3`, concurrent writes, or events on different nodes. Extraction
follows write-batch iteration order, but clients must not depend on cross-key
ordering.
- The event stream is a cache-invalidation hint, not a reliable CDC mechanism.
## 6. Tests
Add focused coverage, grouped by concern.
Configuration:
- Flag parsing, rejection of unsupported or invalid flags, and `CONFIG SET`
runtime changes.
- Config bitmask: accepted flags map to distinct bits separate from the
existing `RedisType` enum, and `g` / `$` make `del` / `set` pass the filter for
their event classes.
- `A` expansion sets the accepted Kvrocks data-class bits (`g$lshzt`) and makes
`KEA` pass `set` / `del` through filtering.
Emission:
- `SET` publishes `set` only on success; `NX` / `XX` condition failures publish
nothing.
- `DEL` publishes one `del` per actually deleted key, none for missing keys.
- `UNLINK`, `APPEND`, `SETRANGE`, `INCR`, and expiry paths publish no
notifications.
- Disabled configuration emits nothing; replica subscribers receive `set` /
`del` from master writes.
LogData and replication:
- Write policy: with primary class bits off, supported commands add no
notification command context and replicas have nothing to publish; with them
on, the extended `WriteBatchLogData` context is present.
- Codec: existing `WriteBatchLogData` still decodes Redis type and arguments;
new notification command codes for `set` / `del` are recognized, while unknown
command context is skipped by the notification handler without aborting
`WriteBatch::Iterate`.
- WAL compatibility: handlers that only understand existing Redis type LogData
continue to iterate and apply a batch containing the extended arguments.
- Key extraction: notification keys are decoded from metadata-column-family
`PutCF` / `DeleteCF` records, not from raw key names stored in LogData
arguments.
- A malformed notification context on a replica is logged/skipped without
blocking replication progress or ack.
Semantics:
- Duplicate prevention: one successful primary write yields exactly one
publication, not command-layer plus WAL-replay.
- Crash recovery: WAL recovery republishes no notifications; apply-time
publication is best-effort and not replayed after restart.
- Runtime race: flags changed by `CONFIG SET` between LogData-context write and
publication take effect at publish time.
- Ordering: keyspace precedes keyevent for one affected key; no test relies on
cross-key ordering.
- Namespace mapping: default maps to `0`; non-default names are prefixed and
percent-encoded, including numeric names like `"0"`, so tenant channels cannot
collide.
## 7. References
- [#2915](https://github.com/apache/kvrocks/issues/2915) — feature request
driving this design.
- [Redis Keyspace
Notifications](https://redis.io/docs/latest/develop/use/keyspace-notifications/)
— the compatibility target for channels, payloads, and flags.
GitHub link: https://github.com/apache/kvrocks/discussions/3533
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
