GitHub user Aetherance created a discussion: Proposal: Keyspace Notifications
for Kvrocks
## 1. Background and goals
Applications reading from a read-only Kvrocks replica keep a local cache to cut
latency and reduce round-trips to the primary. Keeping that cache fresh
requires a notification when a key changes. Kvrocks has no such mechanism
today, which blocks applications that depend on Redis Keyspace Notifications
from migrating.
This design implements the minimal Redis-compatible Keyspace Notifications
surface needed by #2915:
- `notify-keyspace-events` configuration.
- Events published to `__keyspace@<db>__:<key>` with payload = event name.
- Events published to `__keyevent@<db>__:<event>` with payload = key.
- Consumption through the existing `SUBSCRIBE` / `PSUBSCRIBE`.
- `SET` and `DEL` notifications on both primary and replica.
The MVP is intentionally narrow. It does not infer full Redis command semantics
from RocksDB storage mutations: those records cannot distinguish `SET` from
`APPEND`, `SETRANGE`, or `INCR`, nor a genuine `DEL` from lazy expiry or
internal delete-and-recreate flows.
## 2. Architecture
Redis emits keyspace events at the command execution point, and replicas
re-emit them by replaying the replicated command stream. Kvrocks replicates the
RocksDB WAL (`WriteBatch`), not Redis commands, so replicas cannot recover
exact command semantics from storage mutations alone.
The design therefore stores explicit keyspace event markers in the same
`WriteBatch` as the data mutation:
```mermaid
flowchart TD
subgraph primary["Primary / standalone"]
cmd["Command layer<br/>SET / DEL succeeds"]
marker["Append event marker<br/>to the same WriteBatch"]
write["db_->Write(batch) succeeds"]
decode_primary["Decode markers<br/>from the batch"]
notify_primary["NotifyKeyspaceEvent"]
end
subgraph replica["Replica"]
recv["Receive replicated WriteBatch"]
apply["Apply batch succeeds"]
decode_replica["Decode markers<br/>from the same batch"]
notify_replica["NotifyKeyspaceEvent"]
end
pub["PublishMessage<br/>keyspace / keyevent"]
clients["Local SUBSCRIBE / PSUBSCRIBE subscribers"]
cmd --> marker --> write --> decode_primary --> notify_primary
marker -. "replicated with WriteBatch" .-> recv
recv --> apply --> decode_replica --> notify_replica
notify_primary --> pub
notify_replica --> pub
pub --> clients
```
This keeps the WAL as the replication carrier without guessing event names from
column-family writes. The primary publishes after `db_->Write` succeeds; the
replica publishes after it applies the same batch. With the same local
`notify-keyspace-events` setting, both roles publish the same marker-derived
events.
Pub/Sub fan-out is reused unchanged: `Server::PublishMessage` already handles
exact-channel and pattern subscriptions and is a pure in-memory operation, so
it works on read-only replicas.
## 3. Design decisions
### 3.1 Config: `notify-keyspace-events`
Add the `notify-keyspace-events` config. The default is `""` (disabled). The
parser accepts the Redis 7 flag character set `K E A g $ l s h z x e t m n d`,
rejects unknown characters, and supports runtime updates through `CONFIG SET`.
The MVP accepts the full flag set for compatibility, but only `g` (`del`) and
`$` (`set`) currently produce notifications. Internally these flags are parsed
into non-overlapping bits:
| Constant | Bit | Flag | Meaning |
|---|---:|---|---|
| `kKeyspace` | `1 << 0` | `K` | publish keyspace channels |
| `kKeyEvent` | `1 << 1` | `E` | publish keyevent channels |
| `kGeneric` | `1 << 2` | `g` | generic events; MVP emits `del` |
| `kString` | `1 << 3` | `$` | string events; MVP emits `set` |
| `kList` | `1 << 4` | `l` | accepted, no MVP emitter |
| `kSet` | `1 << 5` | `s` | accepted, no MVP emitter |
| `kHash` | `1 << 6` | `h` | accepted, no MVP emitter |
| `kZSet` | `1 << 7` | `z` | accepted, no MVP emitter |
| `kExpired` | `1 << 8` | `x` | accepted, no MVP emitter |
| `kEvicted` | `1 << 9` | `e` | accepted, no MVP emitter |
| `kStream` | `1 << 10` | `t` | accepted, no MVP emitter |
| `kKeyMiss` | `1 << 11` | `m` | accepted, no MVP emitter |
| `kNew` | `1 << 12` | `n` | accepted, no MVP emitter |
| `kModule` | `1 << 13` | `d` | accepted, no MVP emitter |
`A` expands to `g$lshzxetd` for Redis compatibility. It does not include `m`
(`keymiss`) or `n` (`new`), which must be enabled explicitly. In this MVP, only
`$` and `g` produce events, so the common `KEA` setting enables both channel
forms and lets `set` / `del` pass the filter.
The other classes enabled by `A` are accepted but inert for now. If later
releases add markers for list/hash/etc., existing `KEA` deployments will
naturally start receiving those events; that matches Redis semantics, but
should be called out in release notes.
### 3.2 Event marker format
Add a keyspace-event log-data record that can be appended to a RocksDB
`WriteBatch`:
```cpp
struct KeyspaceEventLogData {
int type_class; // KeyspaceEvent::kString or KeyspaceEvent::kGeneric
std::string event; // "set" or "del"
std::string ns; // Kvrocks namespace
std::string key; // user key
};
```
The record is encoded with `WriteBatch::PutLogData()` using a versioned,
non-numeric binary tag:
```text
"kse1" // 4-byte ASCII tag, leading 'k'
uint8 type_class_code // 1 = kString, 2 = kGeneric
uint8 event_code // 1 = set, 2 = del
uint32 namespace_len_le
bytes namespace
uint32 key_len_le
bytes key
```
`type_class_code` and `event_code` use compact one-byte encodings. The decoder
maps them back to the event class and event name before calling the emitter.
Markers use an independent `kse1` tag instead of reusing normal
`redis::WriteBatchLogData`. This avoids confusion with existing Redis type logs
and lets older readers treat unknown markers as skippable log data rather than
command-reconstruction context.
Markers are self-contained: each record carries its own event class, event
name, namespace, and key. The decoder does not infer the affected key from
neighboring `PutCF` / `DeleteCF` records or depend on marker position inside
the batch. Writers may place a marker after the mutation it describes for
debuggability, but correctness does not rely on that order.
Unknown versions, unknown codes, truncated payloads, and malformed markers are
skipped after at most a warning; they must not fail an already committed write
or block replication. During rolling upgrades, older nodes may ignore `kse1`
markers, but they must still apply and continue replicating batches that
contain them.
Markers are semantic facts, not publication decisions. They are written when a
command has definitely performed a supported mutation. Each node still applies
its own `notify-keyspace-events` filter when publishing.
Marker write policy:
- If the primary's `notify-keyspace-events` enables no supported class bit (`g`
/ `$`), supported commands may skip writing markers to avoid default-off
overhead. The skip keys only on class bits, never on the `K` / `E` channel
selectors, so a primary with a class bit but no selector still writes markers
for replicas to publish.
- If the primary skips markers, replicas cannot publish notifications for that
write even if their local config is later enabled (the operational consequence
for replica caches is stated as a guarantee in §5).
- When enabled, each supported `SET` / deleted key adds one small log-data
record to the WAL and replication stream. This is the cost of preserving
command semantics for replicas.
Configuration is thus checked twice — at marker-write time on the primary and
at publish time on each node; a `CONFIG SET` that disables notifications
between the two still ships the marker to replicas but suppresses the primary's
local publish.
### 3.3 Marker writers
Only commands with unambiguous MVP semantics write markers:
- `SET`: append a `set` marker only when the command actually writes the key.
Conditional variants such as `SET key value NX` must not write a marker when
the condition fails.
- `DEL`: append one `del` marker per key actually deleted, each carrying that
single key. Missing keys do not produce markers.
Every other write path (the out-of-scope list in §4) writes no markers in this
MVP. This is deliberate: emitting no event is safer than emitting a Redis event
name that does not match the command semantics.
### 3.4 Event decoder and trigger points
Primary and replica use the same marker decoder, but invoke it at different
points:
- The primary decodes the just-written batch after `db_->Write` succeeds, then
calls `Server::NotifyKeyspaceEvent`.
- The replica applies the replicated batch first, decodes that same batch, then
calls `Server::NotifyKeyspaceEvent`.
The command layer writes markers but never publishes directly. This keeps
failed writes silent and prevents a single-node write from publishing once from
the command path and once from the batch path.
Publication is bound only to live write/apply paths. During restart, RocksDB
replays WAL internally inside `DB::Open`; that recovery does not invoke the
notification path, so historical markers are not republished.
Decode errors are non-fatal. A malformed marker is logged and skipped; keyspace
notifications must never abort a committed write or stall replication.
### 3.5 Event emitter
`Server::NotifyKeyspaceEvent` applies the local config filter and reuses the
existing Pub/Sub path to publish the two Redis-compatible channel forms:
For a `SET foo bar` marker in the default namespace, `KEA` publishes:
```text
__keyspace@0__:foo -> "set"
__keyevent@0__:set -> "foo"
```
For a successful `DEL foo`, it publishes:
```text
__keyspace@0__:foo -> "del"
__keyevent@0__:del -> "foo"
```
### 3.6 `<db>` / namespace mapping
For Redis compatibility, the default namespace maps to db `0`, so standard
subscriptions such as `PSUBSCRIBE __keyevent@0__:*` work out of the box.
Non-default Kvrocks namespaces cannot be inserted into `<db>` directly. A
namespace literally named `"0"` would collide with Redis db `0` channels and
could leak events across tenants. The mapping must therefore be collision-free.
`MapNamespaceToKeyspaceDB(ns)` is defined as:
- default namespace -> `"0"`;
- non-default namespace -> `"ns:" + PercentEncode(ns)`, where percent encoding
escapes every byte outside `[A-Za-z0-9_.-]` and also escapes `%`.
Subscribers to `__keyevent@0__:*` therefore only see the default namespace.
Subscribers that opt into non-default namespaces use the encoded form, for
example `__keyevent@ns:tenantA__:*`. The payload remains only the key name, so
namespace identity comes from the channel.
The key component in `__keyspace@<db>__:<key>` and the payload in
`__keyevent@<db>__:<event>` are the raw user key bytes, matching Redis
behavior. Keys are not percent-encoded, even if they contain `:`, whitespace,
or binary bytes.
## 4. Event scope
Supported MVP events:
| Command | Event class | Event | Emitted when |
|---|---|---|---|
| `SET` | `$` | `set` | the command writes the key |
| `DEL` | `g` | `del` | the key is actually deleted |
Out of scope:
- `APPEND`, `SETRANGE`, `GETSET`, `INCR`, and other string commands.
- Hash/list/set/zset/stream element updates.
- `EXPIRE`, `PEXPIRE`, `expired`, and lazy/compaction expiry.
- `FLUSHDB` / `FLUSHALL` and `DeleteRangeCF` notifications.
- `rename_from`, `rename_to`, `new`, `keymiss`, `evicted`, and module events.
`FLUSHDB` is explicitly a limitation for cache invalidation: this MVP does not
emit per-key or aggregate flush notifications. Applications that rely on flush
visibility need a later design for aggregate `flushdb` / `flushall` events.
## 5. Semantics and guarantees
- Best effort only. Notifications are Pub/Sub messages: they are not persisted,
acknowledged, replayed, or retried.
- Failed writes publish nothing. A marker in an uncommitted batch has no
visible effect because publication only runs after successful `db_->Write` or
replica apply.
- A write commits before its local notification publishes; a crash in that
window can make local subscribers miss the event. Replica notifications publish
only after apply, so they can lag the primary, and a replica crash between
apply and publish loses the event — restart recovery does not replay it, which
also prevents duplicate notifications after crashes.
- `notify-keyspace-events` is node-local, and the primary's setting gates
marker writing (§3.2). A primary with no supported class bit writes no markers,
so its replicas publish nothing no matter how they are configured. For the
#2915 replica-cache use case operators must therefore enable a supported class
bit (for example `KEA`) on the primary — even if it has no local subscribers —
and set the same value on every subscribed replica.
- Rolling upgrades rely on unknown-marker tolerance. Older nodes may ignore the
marker and not publish, but they must still apply and replicate the write batch.
- There is no cross-node de-duplication: a client subscribed to both primary
and replica may process the same logical mutation twice.
- For one marker, keyspace is published before keyevent on one node. No
ordering is promised across different markers — multi-key commands like `DEL k1
k2 k3`, concurrent writes, or events on different nodes. Marker order inside a
batch is not part of the contract; each marker is self-contained.
- The event stream is a cache-invalidation hint, not a reliable CDC mechanism.
## 6. Tests
Add focused coverage, grouped by concern.
Configuration:
- Flag parsing, invalid-flag rejection, and `CONFIG SET` runtime changes.
- Bit assignments: `K`, `E`, `g`, `$` map to distinct bits matching marker
`type_class` values.
- `A` expansion sets the Redis all-events class bits (`g$lshzxetd`), excludes
`m` / `n`, and makes `KEA` pass `set` / `del` through filtering.
Emission:
- `SET` publishes `set` only on success; `NX` / `XX` condition failures publish
nothing.
- `DEL` publishes one `del` per actually deleted key, none for missing keys.
- `APPEND`, `SETRANGE`, `INCR`, and expiry paths publish no markers.
- Disabled configuration emits nothing; replica subscribers receive `set` /
`del` from master writes.
Markers and replication:
- Write policy: with primary class bits off, supported commands add no marker
overhead and replicas have nothing to publish; with them on, markers are
present.
- Codec: valid `set` / `del` markers round-trip; unknown versions, unknown
codes, and truncated payloads are skipped without aborting
`WriteBatch::Iterate`.
- WAL compatibility: a handler that does not recognize `kse1` still iterates
and applies a batch containing the marker.
- Self-containment: a marker not adjacent to its mutation still decodes from
its own payload.
- A malformed marker on a replica is logged/skipped without blocking
replication progress or ack.
Semantics:
- Duplicate prevention: one successful primary write yields exactly one
publication, not command-layer plus WAL-replay.
- Crash recovery: WAL recovery republishes no markers; apply-time publication
is best-effort and not replayed after restart.
- Runtime race: flags changed by `CONFIG SET` between marker write and
publication take effect at publish time.
- Ordering: keyspace precedes keyevent for one marker; no test relies on
cross-marker ordering.
- Namespace mapping: default maps to `0`; non-default names are prefixed and
percent-encoded, including numeric names like `"0"`, so tenant channels cannot
collide.
## 7. References
- [#2915](https://github.com/apache/kvrocks/issues/2915) — feature request
driving this design.
- [Redis Keyspace
Notifications](https://redis.io/docs/latest/develop/use/keyspace-notifications/)
— the compatibility target for channels, payloads, and flags.
GitHub link: https://github.com/apache/kvrocks/discussions/3533
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
