GitHub user bitflicker64 edited a discussion: Cross-server schema cache 
consistency without broadcast storms (#2617)

## Scope and current status

Updated 2026-09-18. This discussion tracks **cross-server schema-cache 
convergence**, including missed notifications and excessive invalidation 
traffic. Restoring a watch connection is necessary, but does not by itself make 
schema caches consistent.

A schema change on Server A can remain invisible on Server B because the 
publisher emits no notification, the watch stops, an event is lost during 
disconnection, or the consumer fails to invalidate its cache. These are 
different failure points in the same propagation path.

```mermaid
flowchart LR
    A["Server A: schema mutation"] --> B["Publish invalidation"]
    B --> C["PD watch"]
    C --> D["Server B: invalidate cache"]
    V["Proposed durable schema version"] -. "periodic reconciliation" .-> D
    A -. "proposed version update" .-> V
```

- **#3157 repairs watch recovery**, the transport segment in this diagram. It 
does not add event replay, missing schema notifications, or cache 
reconciliation.
- **This discussion covers the end-to-end schema contract.** The 
version/reconciliation path is a proposal, not shipped behavior or an approved 
release requirement.
- **#3151 tracks graph metadata divergence**, a related but distinct scope: 
graph configuration/lifecycle visibility across Servers. Share recovery 
principles without treating graph lifecycle and schema-cache invalidation as 
interchangeable.

## Timeline and related work

Dates below use UTC; merged dates are distinguished from issue reports and 
review updates.

| Date | Item | What changed / what remains |
|---|---|---|
| 2024-08-06 | [#2617](https://github.com/apache/hugegraph/issues/2617), 
reported; now closed | Multi-server HStore schema visibility depended on which 
Server answered. The historical issue being closed does not establish complete 
cache convergence. |
| 2026-05-06 | [#3011](https://github.com/apache/hugegraph/pull/3011), merged | 
Added cross-server HStore schema-cache clear propagation. Updates/status 
changes and conditional removal notification remain relevant here. |
| 2026-05-15 | [#3017](https://github.com/apache/hugegraph/pull/3017), merged | 
Aligned actions on the local EventHub path; this is not a cross-server 
reconciliation mechanism. |
| 2026-06-23 | [#3062](https://github.com/apache/hugegraph/pull/3062), merged | 
Restored terminal-watch recovery for the **etcd** meta driver. A fresh 
subscription still does not replay the disconnected interval; this is a 
separate driver from PD KvClient. |
| 2026-08-12 | [#3138](https://github.com/apache/hugegraph/pull/3138), merged | 
Closed the creating Server's local graph-binding race. Other replicas' graph 
readiness remains under 
[#3137](https://github.com/apache/hugegraph/issues/3137). |
| 2026-08-12 | [#3151](https://github.com/apache/hugegraph/issues/3151) / 
[#3152](https://github.com/apache/hugegraph/issues/3152), opened | Separated 
missed graph-metadata events from PD watch permanently stopping after reconnect 
failure. Both remain open at this update. |
| 2026-08-15 | [#3157](https://github.com/apache/hugegraph/pull/3157), opened | 
Per-subscription PD watch recovery, stale-callback rejection, and separation of 
transport cleanup from client shutdown. |
| 2026-09-17 | [#3157 review and 
fixes](https://github.com/apache/hugegraph/pull/3157#pullrequestreview-5237464442)
 | `eeb7041e8` preserves ordinary KV/lock timeout budgets and unpublishes 
SchemaDriver before cleanup. Five independent review lanes; 41 KvClient + 5 
SchemaDriver tests passed. |
| 2026-09-18 | #3157 status checkpoint | Still **open**, head `eeb7041e8`; 24 
CI checks passed. One legacy streaming override compatibility discussion 
remains. No event replay or reconciliation was added. |

**Recommended order:** land the bounded watch-recovery fix in #3157 
independently; verify and address the small publisher/consumer defects below; 
then agree and implement the convergence contract in this discussion. Do not 
expand #3157 into the full schema synchronization redesign, and do not close 
this discussion merely because #3157 merges.

## Background

The remaining gap exists even with a healthy watch. 
`CachedSchemaTransactionV2.updateSchema()` updates its local cache but 
suppresses cross-server notification to avoid broadcast storms from background 
status changes. Removal notification is gated by `task.sync_deletion`. 
Reconnecting cannot deliver a notification that was never emitted, nor recover 
an event from before the fresh subscription.

Multiple Server pods behind one Kubernetes Service expose this as inconsistent 
schema visibility depending on the responding pod. The proposal below aims to 
bound stale caches without turning every mutation into a full cluster-wide 
reload.

## Why the quick fix does not work

Re-enabling the suppressed notification looks like a one-line fix. The numbers 
say otherwise. An index rebuild flips status 2 to 3 times per label 
(`IndexLabelRebuildJob.java:106,137,143`). Take M = 1,000 status flips over 60 
s on N = 10 servers with 5,000 schema elements per graph:

- 1,000 event puts become 10,000 watch deliveries;
- each delivery is a full schema-cache clear (`clearSchemaCache`, 
`CachedSchemaTransactionV2.java:131-152`), so 10,000 full clears;
- every cleared server lazily reloads all four schema types, and the V2 
name-miss path runs a full reload inline on the request thread (`:414-420`), so 
concurrent readers pile up: roughly 40,000 prefix scans for the duration of the 
job, with latency spikes on every server.

Cluster-wide work scales as O(M x N). Any acceptable design has to make 
cluster-wide work independent of M. The same math applies to today's shipped 
add path: bulk-creating 500 schema elements already produces 500 uncoalesced 
events and 500 full clears per remote server.

## What propagates today

| Mode | add | update / status | remove | transport |
|---|---|---|---|---|
| hstore + PD (V2) | yes, one event per op | no, silent | gated off when 
`task.sync_deletion=true` | one PD key, full clear per event |
| HBase, N servers (V1) | local JVM only | local only | local only | RPC 
notifier exists but is off by default and needs a static peer list |
| rocksdb + raft | covered | covered | covered | raft log replay |
| single node | covered (#3017) | n/a | n/a | in-process EventHub |

Two details make the multi-server rows worse than they look. The 
`task.sync_deletion` gate (`maybeNotifySchemaCacheClear`, 
`CachedSchemaTransactionV2.java:360-367`) assumes the caller invalidates other 
nodes synchronously, but `syncWait` only waits for the local task, so a removed 
label can survive in remote caches indefinitely. And the PD watch client 
re-subscribes after a leader change or error with no resume point 
(`KvClient.java:~168-190`), so any design whose correctness depends on event 
delivery is wrong by construction.

## Proposed design: three rules

The historical tension exists because the event is the only carrier of truth: 
losing one loses the information forever, so nothing dared coalesce or drop 
them. The proposal separates truth from wake-up.

![Schema change propagation, current path and proposed 
path](https://raw.githubusercontent.com/bitflicker64/hugegraph/images/2617/.github/images/schema-sync-2617.svg)

**Rule 1: a version register, written synchronously on every mutation.** Each 
graph gets a small version value stored next to the schema itself: a PD KV key 
for hstore (`HUGEGRAPH/{cluster}/SCHEMA_VERSION/{graphSpace}/{graph}`, an 
opaque token compared only for equality), a counter row on the existing counter 
table for HBase (no DDL, no migration). Every schema mutation, including 
updates and status flips, writes it on the DDL thread right after the schema 
write. One cheap put with zero fan-out; the storm was fan-out times full 
reloads, never the publisher's own puts. If the register put fails after the 
schema commit, the DDL call still succeeds, and a background retry with metrics 
(`register_write_failures`, `register_dirty`) takes over until it lands.

**Rule 2: events become coalesced, lossy hints.** A debounced flusher (500 ms 
window) publishes per-id invalidation hints on a new `EVENT/GRAPH/SCHEMA/DELTA` 
key, at most about 2 events per second per graph no matter how fast mutations 
come. Hints only invalidate cache entries; they can never mark a node as up to 
date. That makes dropping, merging and reordering them harmless, which is 
exactly what allows unlimited coalescing. For compatibility, bursts containing 
add/remove/clear also put one event per window on the legacy CLEAR key, 
byte-identical to today's payload, so old servers and store nodes see what they 
see today, only fewer of them.

**Rule 3: a node advances its applied state only through full resync.** A 
reconciler polls the register (default every 10 s per graph, one prefix scan 
per server per tick) and, on mismatch, clears the graph's caches and adopts the 
value it read before clearing. Nothing else advances the applied state, not 
even the hints. The proposed target is convergence within two successful 
reconcile intervals (20 s with a 10 s interval), after the version update is 
durably visible and under bounded read/reload latency. This is not an 
unconditional deadline during PD unavailability. A schema commit followed by a 
failed version write and publisher crash can lose an in-memory retry; durable 
recovery or an atomic publication contract must be established before claiming 
a bound after every successful DDL. Typical hint-path latency also needs 
measurement.

For the storm scenario above, this comes out to at most 121 events on the wire 
and about 40 full clears cluster-wide instead of 10,000, a roughly 250x 
reduction, and doubling M changes nothing on the receiver side. Schema read hit 
paths gain zero instructions; the register is only touched on writes and by the 
reconciler.

![The same 1,000-flip burst on a timeline: register moves per mutation, hints 
stay under 121 events, resyncs stay under 4 per 
server](https://raw.githubusercontent.com/bitflicker64/hugegraph/images/2617/.github/images/burst-timeline-2617.svg)

## Proposed convergence targets, per deployment

| Mode | staleness bound after a mutation | config needed |
|---|---|---|
| hstore + PD, N pods | Hint delivery for the fast path; target two successful 
reconcile intervals after durable version publication | Proposed on by default; 
not implemented |
| HBase, N servers | Poll-based convergence target; optional scope still 
undecided | See Q1; not implemented |
| rocksdb + raft | unchanged (raft already covers it) | none |
| single node | unchanged, zero | none |

An emergency switch (`schema.sync.enabled=false`) restores the legacy event 
behavior. Rolling-upgrade compatibility is a design goal: keep legacy payloads 
readable, then validate mixed-version ordering, coalescing delays and recovery 
before claiming safety in any pod order.

![Who publishes and watches which event key during a rolling upgrade: the new 
DELTA key is invisible to old servers and store 
nodes](https://raw.githubusercontent.com/bitflicker64/hugegraph/images/2617/.github/images/wire-topology-2617.svg)

## Candidate preparatory fixes to verify independently

The original analysis identified the following publisher/consumer defects or 
optimization candidates. Recheck each on the target head before changing code; 
#3157 does not claim to fix them. The JSON payload item concerns Store-side 
SchemaDriver, not proof that a Server watch failed:

1. Store nodes mis-parse every schema-clear event today: 
`SchemaDriver.schemaCacheClearHandler` (`SchemaDriver.java:137-147`) still 
splits the payload on `-`, but the server has emitted JSON since #3011, so 
store-side schema caches silently never clear.
2. `SchemaTransactionV2.saveSchema` publishes 
`notifyGraphVertexCacheClear`/`notifyGraphEdgeCacheClear` PD events on every 
VertexLabel/EdgeLabel save, and nothing anywhere subscribes to them: pure write 
amplification inside the schema write path.
3. The `task.sync_deletion` gate described above, which also suppresses 
STORE_CLEAR/TRUNCATE propagation.
4. `PdMetaDriver.extractValuesFromResponse` returns null for a watch batch 
containing any non-Put event, swallowing the whole batch for every meta 
listener in the process.
5. The V2 name-miss path reloads the entire schema inline on the request 
thread, so concurrent misses after a clear stampede the backend.

## Who this pings, and why

- @imbajin raised the constraint this design is built around (no broadcast 
storms) and most open decisions need a maintainer call.
- @dpol1 wrote #3011, #3017 and #3062; this design extends and partly replaces 
that machinery, including the `task.sync_deletion` gate, so review from that 
side matters most.
- @haohao0103 reported #2617 from a real deployment and is best placed to 
confirm the failure mode described here matches what was observed.
- @Tsukilc has the most recent commits on the V2 cache and meta code and wrote 
`SchemaDriver` in hugegraph-struct, where bug 1 lives.
- @koi2000 for PD watch semantics (question 4 in the comment below).
- @Soyaazz for the store release question (question 7).
- @VGalaxies wrote much of `SchemaTransactionV2`/`MetaManager` and may know 
history that invalidates parts of this.
- @contrueCT for the graph-data cache side, which this proposal deliberately 
defers.
- @vaijosh for whether HBase multi-server deserves the optional mode at all 
(question 1).

Nothing here is settled. If a simpler mechanism reaches the same bounds, or the 
bounds themselves are wrong for the k8s release, that is exactly the feedback 
this thread is for. The full analysis, per-mode matrix, failure semantics and 
test plan exist as a longer document; the open questions and the implementation 
plan are in the first comment below.

Implementation decisions and the proposed phased plan are tracked in the [first 
comment](https://github.com/apache/hugegraph/discussions/3205#discussioncomment-18431491).
 Existing diagrams illustrate the proposal; timing/count annotations are model 
estimates, not measured production guarantees.


GitHub link: https://github.com/apache/hugegraph/discussions/3205

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to