GitHub user bitflicker64 edited a discussion: Cross-server schema cache
consistency without broadcast storms
## Scope and current status
Updated 2026-09-18. This discussion tracks **cross-server schema-cache
convergence**, including missed notifications and excessive invalidation
traffic. Restoring a watch connection is necessary, but does not by itself make
schema caches consistent.
A schema change on Server A can remain invisible on Server B because the
publisher emits no notification, the watch stops, an event is lost during
disconnection, or the consumer fails to invalidate its cache. These are
different failure points in the same propagation path.
```mermaid
flowchart LR
A["Server A: schema mutation"] --> B["Publish invalidation"]
B --> C["PD watch"]
C --> D["Server B: invalidate cache"]
V["Proposed durable schema version"] -. "periodic reconciliation" .-> D
A -. "proposed version update" .-> V
```
- **#3157 repairs watch recovery**, the transport segment in this diagram. It
does not add event replay, missing schema notifications, or cache
reconciliation.
- **This discussion covers the end-to-end schema contract.** The
version/reconciliation path is a proposal, not shipped behavior or an approved
release requirement.
- **#3151 tracks graph metadata divergence**, a related but distinct scope:
graph configuration/lifecycle visibility across Servers. Share recovery
principles without treating graph lifecycle and schema-cache invalidation as
interchangeable.
## Timeline and related work
Dates below use UTC; merged dates are distinguished from issue reports and
review updates.
| Date | Item | What changed / what remains |
|---|---|---|
| 2024-08-06 | [#2617](https://github.com/apache/hugegraph/issues/2617),
reported; now closed | Multi-server HStore schema visibility depended on which
Server answered. The historical issue being closed does not establish complete
cache convergence. |
| 2026-05-06 | [#3011](https://github.com/apache/hugegraph/pull/3011), merged |
Added cross-server HStore schema-cache clear propagation. Updates/status
changes and conditional removal notification remain relevant here. |
| 2026-05-15 | [#3017](https://github.com/apache/hugegraph/pull/3017), merged |
Aligned actions on the local EventHub path; this is not a cross-server
reconciliation mechanism. |
| 2026-06-23 | [#3062](https://github.com/apache/hugegraph/pull/3062), merged |
Restored terminal-watch recovery for the **etcd** meta driver. A fresh
subscription still does not replay the disconnected interval; this is a
separate driver from PD KvClient. |
| 2026-08-12 | [#3138](https://github.com/apache/hugegraph/pull/3138), merged |
Closed the creating Server's local graph-binding race. Other replicas' graph
readiness remains under
[#3137](https://github.com/apache/hugegraph/issues/3137). |
| 2026-08-12 | [#3151](https://github.com/apache/hugegraph/issues/3151) /
[#3152](https://github.com/apache/hugegraph/issues/3152), opened | Separated
missed graph-metadata events from PD watch permanently stopping after reconnect
failure. Both remain open at this update. |
| 2026-08-15 | [#3157](https://github.com/apache/hugegraph/pull/3157), opened |
Per-subscription PD watch recovery, stale-callback rejection, and separation of
transport cleanup from client shutdown. |
| 2026-09-17 | [#3157 review and
fixes](https://github.com/apache/hugegraph/pull/3157#pullrequestreview-5237464442)
| `eeb7041e8` preserves ordinary KV/lock timeout budgets and unpublishes
SchemaDriver before cleanup. Five independent review lanes; 41 KvClient + 5
SchemaDriver tests passed. |
| 2026-09-18 | #3157 status checkpoint | Still **open**, head `eeb7041e8`; 24
CI checks passed. One legacy streaming override compatibility discussion
remains. No event replay or reconciliation was added. |
**Recommended order:** land the bounded watch-recovery fix in #3157
independently; verify and address the small publisher/consumer defects below;
then agree and implement the convergence contract in this discussion. Do not
expand #3157 into the full schema synchronization redesign, and do not close
this discussion merely because #3157 merges.
## Background
The remaining gap exists even with a healthy watch.
`CachedSchemaTransactionV2.updateSchema()` updates its local cache but
suppresses cross-server notification to avoid broadcast storms from background
status changes. Removal notification is gated by `task.sync_deletion`.
Reconnecting cannot deliver a notification that was never emitted, nor recover
an event from before the fresh subscription.
Multiple Server pods behind one Kubernetes Service expose this as inconsistent
schema visibility depending on the responding pod. The proposal below aims to
bound stale caches without turning every mutation into a full cluster-wide
reload.
## Why the quick fix does not work
Re-enabling the suppressed notification looks like a one-line fix. The numbers
say otherwise. An index rebuild flips status 2 to 3 times per label
(`IndexLabelRebuildJob.java:106,137,143`). Take M = 1,000 status flips over 60
s on N = 10 servers with 5,000 schema elements per graph:
- 1,000 event puts become 10,000 watch deliveries;
- each delivery is a full schema-cache clear (`clearSchemaCache`,
`CachedSchemaTransactionV2.java:131-152`), so 10,000 full clears;
- every cleared server lazily reloads all four schema types, and the V2
name-miss path runs a full reload inline on the request thread (`:414-420`), so
concurrent readers pile up: roughly 40,000 prefix scans for the duration of the
job, with latency spikes on every server.
Cluster-wide work scales as O(M x N). Any acceptable design has to make
cluster-wide work independent of M. The same math applies to today's shipped
add path: bulk-creating 500 schema elements already produces 500 uncoalesced
events and 500 full clears per remote server.
## What propagates today
| Mode | add | update / status | remove | transport |
|---|---|---|---|---|
| hstore + PD (V2) | yes, one event per op | no, silent | gated off when
`task.sync_deletion=true` | one PD key, full clear per event |
| HBase, N servers (V1) | local JVM only | local only | local only | RPC
notifier exists but is off by default and needs a static peer list |
| rocksdb + raft | covered | covered | covered | raft log replay |
| single node | covered (#3017) | n/a | n/a | in-process EventHub |
Two details make the multi-server rows worse than they look. The
`task.sync_deletion` gate (`maybeNotifySchemaCacheClear`,
`CachedSchemaTransactionV2.java:360-367`) assumes the caller invalidates other
nodes synchronously, but `syncWait` only waits for the local task, so a removed
label can survive in remote caches indefinitely. And the PD watch client
re-subscribes after a leader change or error with no resume point
(`KvClient.java:~168-190`), so any design whose correctness depends on event
delivery is wrong by construction.
## Proposed design: three rules
The historical tension exists because the event is the only carrier of truth:
losing one loses the information forever, so nothing dared coalesce or drop
them. The proposal separates truth from wake-up.

**Rule 1: a version register, written synchronously on every mutation.** Each
graph gets a small version value stored next to the schema itself: a PD KV key
for hstore (`HUGEGRAPH/{cluster}/SCHEMA_VERSION/{graphSpace}/{graph}`, an
opaque token compared only for equality), a counter row on the existing counter
table for HBase (no DDL, no migration). Every schema mutation, including
updates and status flips, writes it on the DDL thread right after the schema
write. One cheap put with zero fan-out; the storm was fan-out times full
reloads, never the publisher's own puts. If the register put fails after the
schema commit, the DDL call still succeeds, and a background retry with metrics
(`register_write_failures`, `register_dirty`) takes over until it lands.
**Rule 2: events become coalesced, lossy hints.** A debounced flusher (500 ms
window) publishes per-id invalidation hints on a new `EVENT/GRAPH/SCHEMA/DELTA`
key, at most about 2 events per second per graph no matter how fast mutations
come. Hints only invalidate cache entries; they can never mark a node as up to
date. That makes dropping, merging and reordering them harmless, which is
exactly what allows unlimited coalescing. For compatibility, bursts containing
add/remove/clear also put one event per window on the legacy CLEAR key,
byte-identical to today's payload, so old servers and store nodes see what they
see today, only fewer of them.
**Rule 3: a node advances its applied state only through full resync.** A
reconciler polls the register (default every 10 s per graph, one prefix scan
per server per tick) and, on mismatch, clears the graph's caches and adopts the
value it read before clearing. Nothing else advances the applied state, not
even the hints. The proposed target is convergence within two successful
reconcile intervals (20 s with a 10 s interval), after the version update is
durably visible and under bounded read/reload latency. This is not an
unconditional deadline during PD unavailability. A schema commit followed by a
failed version write and publisher crash can lose an in-memory retry; durable
recovery or an atomic publication contract must be established before claiming
a bound after every successful DDL. Typical hint-path latency also needs
measurement.
For the storm scenario above, this comes out to at most 121 events on the wire
and about 40 full clears cluster-wide instead of 10,000, a roughly 250x
reduction, and doubling M changes nothing on the receiver side. Schema read hit
paths gain zero instructions; the register is only touched on writes and by the
reconciler.

## Proposed convergence targets, per deployment
| Mode | staleness bound after a mutation | config needed |
|---|---|---|
| hstore + PD, N pods | Hint delivery for the fast path; target two successful
reconcile intervals after durable version publication | Proposed on by default;
not implemented |
| HBase, N servers | Poll-based convergence target; optional scope still
undecided | See Q1; not implemented |
| rocksdb + raft | unchanged (raft already covers it) | none |
| single node | unchanged, zero | none |
An emergency switch (`schema.sync.enabled=false`) restores the legacy event
behavior. Rolling-upgrade compatibility is a design goal: keep legacy payloads
readable, then validate mixed-version ordering, coalescing delays and recovery
before claiming safety in any pod order.

## Candidate preparatory fixes to verify independently
The original analysis identified the following publisher/consumer defects or
optimization candidates. Recheck each on the target head before changing code;
#3157 does not claim to fix them. The JSON payload item concerns Store-side
SchemaDriver, not proof that a Server watch failed:
1. Store nodes mis-parse every schema-clear event today:
`SchemaDriver.schemaCacheClearHandler` (`SchemaDriver.java:137-147`) still
splits the payload on `-`, but the server has emitted JSON since #3011, so
store-side schema caches silently never clear.
2. `SchemaTransactionV2.saveSchema` publishes
`notifyGraphVertexCacheClear`/`notifyGraphEdgeCacheClear` PD events on every
VertexLabel/EdgeLabel save, and nothing anywhere subscribes to them: pure write
amplification inside the schema write path.
3. The `task.sync_deletion` gate described above, which also suppresses
STORE_CLEAR/TRUNCATE propagation.
4. `PdMetaDriver.extractValuesFromResponse` returns null for a watch batch
containing any non-Put event, swallowing the whole batch for every meta
listener in the process.
5. The V2 name-miss path reloads the entire schema inline on the request
thread, so concurrent misses after a clear stampede the backend.
## Who this pings, and why
- @imbajin raised the constraint this design is built around (no broadcast
storms) and most open decisions need a maintainer call.
- @dpol1 wrote #3011, #3017 and #3062; this design extends and partly replaces
that machinery, including the `task.sync_deletion` gate, so review from that
side matters most.
- @haohao0103 reported #2617 from a real deployment and is best placed to
confirm the failure mode described here matches what was observed.
- @Tsukilc has the most recent commits on the V2 cache and meta code and wrote
`SchemaDriver` in hugegraph-struct, where bug 1 lives.
- @koi2000 for PD watch semantics (question 4 in the comment below).
- @Soyaazz for the store release question (question 7).
- @VGalaxies wrote much of `SchemaTransactionV2`/`MetaManager` and may know
history that invalidates parts of this.
- @contrueCT for the graph-data cache side, which this proposal deliberately
defers.
- @vaijosh for whether HBase multi-server deserves the optional mode at all
(question 1).
Nothing here is settled. If a simpler mechanism reaches the same bounds, or the
bounds themselves are wrong for the k8s release, that is exactly the feedback
this thread is for. The full analysis, per-mode matrix, failure semantics and
test plan exist as a longer document; the open questions and the implementation
plan are in the first comment below.
Implementation decisions and the proposed phased plan are tracked in the [first
comment](https://github.com/apache/hugegraph/discussions/3205#discussioncomment-18431491).
Existing diagrams illustrate the proposal; timing/count annotations are model
estimates, not measured production guarantees.
GitHub link: https://github.com/apache/hugegraph/discussions/3205
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]