GitHub user bitflicker64 created a discussion: Cross-server schema cache 
consistency without broadcast storms (#2617)


## Background

When a schema change lands on one server, other servers in the same cluster can 
keep serving the old schema from their local caches. #2617 reported this from a 
multi-server HStore deployment: schema created through one server shows up 
intermittently in Hubble, depending on which server answers. Two fixes landed 
since: #3017 made the event path inside a single server consistent, and #3011 
added cross-server clear events, but only on the hstore path and only for add 
and remove. Schema updates and status changes still never reach other servers, 
and the k8s/helm topology (several server pods behind one service) hits this on 
every deployment.

The obvious question is why updates were left out. The comment at 
`CachedSchemaTransactionV2.java:321-322` answers it: notifying on every 
`updateSchemaStatus()` call from background jobs causes a broadcast storm. So 
the current state is a deliberate trade: silence on updates to avoid storms, at 
the price of unbounded staleness. This post proposes a design that removes the 
trade instead of picking a side, and it is open to challenge in every part.

<details>
<summary>中文</summary>

当 schema 变更落在某一台 server 上时,同集群中的其他 server 可能仍然从本地 cache 中返回旧的 schema。#2617 
报告的正是多 server HStore 部署下的这个问题:通过某一台 server 创建的 schema 在 Hubble 中时有时无,取决于由哪台 
server 来响应请求。此后合入了两个修复:#3017 让单台 server 内部的事件路径变得一致,#3011 增加了跨 server 的 clear 
事件,但只覆盖 hstore 路径,并且只覆盖 add 和 remove。schema 的 update 和状态变更仍然完全不会到达其他 server,而 
k8s/helm 拓扑(一个 service 后面挂多个 server pod)在每次部署中都会遇到这个问题。

一个显而易见的疑问是:为什么 update 被排除在外。`CachedSchemaTransactionV2.java:321-322` 
处的注释给出了答案:后台任务每次调用 `updateSchemaStatus()` 都发通知会造成广播风暴。所以当前的状态是一个有意为之的取舍:对 
update 保持静默以避免风暴,代价是没有上界的陈旧。本文提出的设计不是在两者之间选边,而是把这个取舍本身消除掉,并且其中任何部分都欢迎质疑。

</details>

## Why the quick fix does not work

Re-enabling the suppressed notification looks like a one-line fix. The numbers 
say otherwise. An index rebuild flips status 2 to 3 times per label 
(`IndexLabelRebuildJob.java:106,137,143`). Take M = 1,000 status flips over 60 
s on N = 10 servers with 5,000 schema elements per graph:

- 1,000 event puts become 10,000 watch deliveries;
- each delivery is a full schema-cache clear (`clearSchemaCache`, 
`CachedSchemaTransactionV2.java:131-152`), so 10,000 full clears;
- every cleared server lazily reloads all four schema types, and the V2 
name-miss path runs a full reload inline on the request thread (`:414-420`), so 
concurrent readers pile up: roughly 40,000 prefix scans for the duration of the 
job, with latency spikes on every server.

Cluster-wide work scales as O(M x N). Any acceptable design has to make 
cluster-wide work independent of M. The same math applies to today's shipped 
add path: bulk-creating 500 schema elements already produces 500 uncoalesced 
events and 500 full clears per remote server.

<details>
<summary>中文</summary>

把被抑制的那次通知重新打开,看上去是一行代码的修复。但数字并不支持这种做法。一次 index rebuild 会让每个 label 的 status 翻转 2 
到 3 次(`IndexLabelRebuildJob.java:106,137,143`)。取 M = 1,000 次状态翻转、发生在 60 s 内、N = 
10 台 server、每个 graph 有 5,000 个 schema 元素:

- 1,000 次事件 put 变成 10,000 次 watch 投递;
- 每一次投递都是一次完整的 schema cache 
清空(`clearSchemaCache`,`CachedSchemaTransactionV2.java:131-152`),也就是 10,000 
次全量清空;
- 每台被清空的 server 都会惰性地重新加载全部四种 schema 类型,而 V2 的 name 未命中路径会在请求线程上内联执行一次完整 
reload(`:414-420`),于是并发的读请求不断堆积:在该任务持续期间大约产生 40,000 次前缀扫描,并且每台 server 上都会出现延迟毛刺。

集群整体的工作量按 O(M x N) 增长。任何可接受的设计都必须让集群整体的工作量与 M 无关。同样的算术也适用于今天已经发布的 add 路径:批量创建 
500 个 schema 元素,已经会在每台远端 server 上产生 500 个未合并的事件和 500 次全量清空。

</details>

## What propagates today

| Mode | add | update / status | remove | transport |
|---|---|---|---|---|
| hstore + PD (V2) | yes, one event per op | no, silent | gated off when 
`task.sync_deletion=true` | one PD key, full clear per event |
| HBase, N servers (V1) | local JVM only | local only | local only | RPC 
notifier exists but is off by default and needs a static peer list |
| rocksdb + raft | covered | covered | covered | raft log replay |
| single node | covered (#3017) | n/a | n/a | in-process EventHub |

Two details make the multi-server rows worse than they look. The 
`task.sync_deletion` gate (`maybeNotifySchemaCacheClear`, 
`CachedSchemaTransactionV2.java:360-367`) assumes the caller invalidates other 
nodes synchronously, but `syncWait` only waits for the local task, so a removed 
label can survive in remote caches indefinitely. And the PD watch client 
re-subscribes after a leader change or error with no resume point 
(`KvClient.java:~168-190`), so any design whose correctness depends on event 
delivery is wrong by construction.

<details>
<summary>中文</summary>

(表格见上方英文部分)

有两个细节让多 server 的那几行比表面看上去更糟。`task.sync_deletion` 
开关(`maybeNotifySchemaCacheClear`,`CachedSchemaTransactionV2.java:360-367`)假设调用方会同步地让其他节点失效,但
 `syncWait` 只等待本地任务,因此一个已被删除的 label 可以在远端 cache 中无限期存活下去。另外,PD 的 watch client 在 
leader 切换或出错之后会重新订阅,而没有任何 resume 
点(`KvClient.java:~168-190`),所以任何正确性依赖于事件必达的设计,在构造上就是错的。

</details>

## Proposed design: three rules

The historical tension exists because the event is the only carrier of truth: 
losing one loses the information forever, so nothing dared coalesce or drop 
them. The proposal separates truth from wake-up.

![Schema change propagation, current path and proposed 
path](https://raw.githubusercontent.com/bitflicker64/hugegraph/images/2617/.github/images/schema-sync-2617.svg)

**Rule 1: a version register, written synchronously on every mutation.** Each 
graph gets a small version value stored next to the schema itself: a PD KV key 
for hstore (`HUGEGRAPH/{cluster}/SCHEMA_VERSION/{graphSpace}/{graph}`, an 
opaque token compared only for equality), a counter row on the existing counter 
table for HBase (no DDL, no migration). Every schema mutation, including 
updates and status flips, writes it on the DDL thread right after the schema 
write. One cheap put with zero fan-out; the storm was fan-out times full 
reloads, never the publisher's own puts. If the register put fails after the 
schema commit, the DDL call still succeeds, and a background retry with metrics 
(`register_write_failures`, `register_dirty`) takes over until it lands.

**Rule 2: events become coalesced, lossy hints.** A debounced flusher (500 ms 
window) publishes per-id invalidation hints on a new `EVENT/GRAPH/SCHEMA/DELTA` 
key, at most about 2 events per second per graph no matter how fast mutations 
come. Hints only invalidate cache entries; they can never mark a node as up to 
date. That makes dropping, merging and reordering them harmless, which is 
exactly what allows unlimited coalescing. For compatibility, bursts containing 
add/remove/clear also put one event per window on the legacy CLEAR key, 
byte-identical to today's payload, so old servers and store nodes see what they 
see today, only fewer of them.

**Rule 3: a node advances its applied state only through full resync.** A 
reconciler polls the register (default every 10 s per graph, one prefix scan 
per server per tick) and, on mismatch, clears the graph's caches and adopts the 
value it read before clearing. Nothing else advances the applied state, not 
even the hints. The consequence: worst-case staleness after any completed 
mutation is bounded at 2 x reconcile interval (20 s at defaults) regardless of 
watch health, event loss, PD leader churn, or a publisher crashing right after 
the register write. Typical convergence stays at flush plus watch delivery, 
well under a second.

For the storm scenario above, this comes out to at most 121 events on the wire 
and about 40 full clears cluster-wide instead of 10,000, a roughly 250x 
reduction, and doubling M changes nothing on the receiver side. Schema read hit 
paths gain zero instructions; the register is only touched on writes and by the 
reconciler.

![The same 1,000-flip burst on a timeline: register moves per mutation, hints 
stay under 121 events, resyncs stay under 4 per 
server](https://raw.githubusercontent.com/bitflicker64/hugegraph/images/2617/.github/images/burst-timeline-2617.svg)

<details>
<summary>中文</summary>

历史上的矛盾之所以存在,是因为事件是真相的唯一载体:丢掉一个事件就永久丢掉了这条信息,所以没有人敢去合并或丢弃它们。本提案把真相和唤醒分离开。

**规则 1:一个 version register(版本寄存器),在每次变更时同步写入。** 每个 graph 都有一个与 schema 
本身存放在一起的小版本值:对 hstore 是一个 PD KV 
key(`HUGEGRAPH/{cluster}/SCHEMA_VERSION/{graphSpace}/{graph}`,一个只做相等性比较的不透明 
token),对 HBase 则是已有 counter 表上的一行 counter(无需 DDL,无需迁移)。每一次 schema 变更,包括 update 
和状态翻转,都会在 DDL 线程上紧跟着 schema 写入之后写入它。这是一次廉价的 put,扇出为零;风暴来自扇出乘以全量 
reload,从来不是发布方自己的那几次 put。如果 register 的 put 在 schema 提交之后失败,DDL 
调用仍然成功,由带指标的后台重试(`register_write_failures`、`register_dirty`)接管,直到写入落地为止。

**规则 2:事件变成合并失效提示,允许丢失。** 一个带防抖的 flusher(500 ms 窗口)在新的 
`EVENT/GRAPH/SCHEMA/DELTA` key 上发布按 id 的失效提示,无论变更来得多快,每个 graph 每秒最多约 2 
个事件。提示只能让 cache 条目失效,永远不能把某个节点标记为已经最新。这让丢弃、合并和乱序都变得无害,而这正是可以无限制合并的前提。为了兼容性,包含 
add/remove/clear 的突发(burst)还会在每个窗口内向遗留的 CLEAR key 上 put 一个事件,payload 
与今天逐字节一致,因此旧的 server 和 store 节点看到的仍然是它们今天看到的内容,只是数量更少。

**规则 3:一个节点只能通过全量重同步(resync)来推进自己的已应用状态。** 一个对账器(reconciler)轮询 register(默认每个 
graph 每 10 s 一次,每台 server 每个 tick 做一次前缀扫描),一旦发现不一致,就清空该 graph 的各类 
cache,并采用它在清空之前读到的那个值。除此之外没有任何东西能推进已应用状态,提示也不行。由此得到的结论是:任何一次已完成的变更之后,最坏情况下的陈旧上界是
 2 x 对账间隔(默认配置下为 20 s),与 watch 是否健康、事件是否丢失、PD leader 是否频繁切换,或者发布方是否在写完 register 
之后立刻崩溃都无关。典型的收敛时间仍然是刷新窗口加上 watch 投递,远低于一秒。

对于上文那个风暴场景,结果是网络上最多 121 个事件、集群范围内约 40 次全量清空,而不是 10,000 次,大约减少 250 倍,并且把 M 
翻倍对接收端没有任何影响。schema 的读命中路径不会多出任何一条指令;register 只在写入时以及被对账器访问。

</details>

## What it guarantees, per deployment

| Mode | staleness bound after a mutation | config needed |
|---|---|---|
| hstore + PD, N pods | ms typical, 20 s worst case at defaults | none, on by 
default |
| HBase, N servers | one reconcile interval typical (poll is the transport) | 
none, if this mode ships at all (see Q1) |
| rocksdb + raft | unchanged (raft already covers it) | none |
| single node | unchanged, zero | none |

An emergency switch (`schema.sync.enabled=false`) restores the legacy event 
behavior. Rolling upgrades are safe in any pod order: old servers keep 
receiving the legacy events they receive today, and coverage improves 
monotonically as pods upgrade.

![Who publishes and watches which event key during a rolling upgrade: the new 
DELTA key is invisible to old servers and store 
nodes](https://raw.githubusercontent.com/bitflicker64/hugegraph/images/2617/.github/images/wire-topology-2617.svg)

<details>
<summary>中文</summary>

(表格见上方英文部分)

一个紧急开关(`schema.sync.enabled=false`)可以恢复遗留的事件行为。滚动升级在任意 pod 顺序下都是安全的:旧的 server 
仍然收到它们今天所收到的遗留事件,而覆盖范围随着 pod 逐个升级单调变好。

</details>

## Bugs found along the way

The code review behind this proposal turned up defects that are worth fixing 
regardless of which design wins:

1. Store nodes mis-parse every schema-clear event today: 
`SchemaDriver.schemaCacheClearHandler` (`SchemaDriver.java:137-147`) still 
splits the payload on `-`, but the server has emitted JSON since #3011, so 
store-side schema caches silently never clear.
2. `SchemaTransactionV2.saveSchema` publishes 
`notifyGraphVertexCacheClear`/`notifyGraphEdgeCacheClear` PD events on every 
VertexLabel/EdgeLabel save, and nothing anywhere subscribes to them: pure write 
amplification inside the schema write path.
3. The `task.sync_deletion` gate described above, which also suppresses 
STORE_CLEAR/TRUNCATE propagation.
4. `PdMetaDriver.extractValuesFromResponse` returns null for a watch batch 
containing any non-Put event, swallowing the whole batch for every meta 
listener in the process.
5. The V2 name-miss path reloads the entire schema inline on the request 
thread, so concurrent misses after a clear stampede the backend.

<details>
<summary>中文</summary>

本提案背后的代码审查还发现了若干缺陷,无论最终采用哪种设计,它们都值得修复:

1. store 节点今天会错误解析每一个 schema clear 
事件:`SchemaDriver.schemaCacheClearHandler`(`SchemaDriver.java:137-147`)仍然按 `-` 
切分 payload,但自 #3011 起 server 发出的已经是 JSON,因此 store 侧的 schema cache 
实际上从来不会被清空,而且没有任何报错。
2. `SchemaTransactionV2.saveSchema` 在每次保存 VertexLabel/EdgeLabel 时都会发布 
`notifyGraphVertexCacheClear`/`notifyGraphEdgeCacheClear` 这两个 PD 
事件,而任何地方都没有订阅者:这是 schema 写入路径内部纯粹的写放大。
3. 上文描述的那个 `task.sync_deletion` 开关,它同时还抑制了 STORE_CLEAR/TRUNCATE 的传播。
4. `PdMetaDriver.extractValuesFromResponse` 只要 watch 批次中包含任何一个非 Put 事件就返回 
null,从而吞掉整个批次,影响进程内所有的 meta listener。
5. V2 的 name 未命中路径会在请求线程上内联重新加载整个 schema,因此在一次清空之后,并发的未命中会对后端形成冲击。

</details>

## Who this pings, and why

- @imbajin raised the constraint this design is built around (no broadcast 
storms) and most open decisions need a maintainer call.
- @dpol1 wrote #3011, #3017 and #3062; this design extends and partly replaces 
that machinery, including the `task.sync_deletion` gate, so review from that 
side matters most.
- @haohao0103 reported #2617 from a real deployment and is best placed to 
confirm the failure mode described here matches what was observed.
- @Tsukilc has the most recent commits on the V2 cache and meta code and wrote 
`SchemaDriver` in hugegraph-struct, where bug 1 lives.
- @koi2000 for PD watch semantics (question 4 in the comment below).
- @Soyaazz for the store release question (question 7).
- @VGalaxies wrote much of `SchemaTransactionV2`/`MetaManager` and may know 
history that invalidates parts of this.
- @contrueCT for the graph-data cache side, which this proposal deliberately 
defers.
- @vaijosh for whether HBase multi-server deserves the optional mode at all 
(question 1).

Nothing here is settled. If a simpler mechanism reaches the same bounds, or the 
bounds themselves are wrong for the k8s release, that is exactly the feedback 
this thread is for. The full analysis, per-mode matrix, failure semantics and 
test plan exist as a longer document; the open questions and the implementation 
plan are in the first comment below.

<details>
<summary>中文</summary>

- @imbajin 提出了本设计所围绕的那个约束(不能有广播风暴),而且大多数未决问题需要 maintainer 来拍板。
- @dpol1 编写了 #3011、#3017 和 #3062;本设计扩展并部分替换了那套机制,其中包括 `task.sync_deletion` 
开关,因此来自这一侧的 review 最为关键。
- @haohao0103 从真实部署中报告了 #2617,最有条件确认这里描述的失效模式是否与当时观察到的现象一致。
- @Tsukilc 在 V2 cache 和 meta 相关代码上有最近的提交,并且编写了 hugegraph-struct 中的 
`SchemaDriver`,也就是 bug 1 所在的位置。
- @koi2000 请看 PD watch 语义(下方评论中的问题 4)。
- @Soyaazz 请看 store 发版的问题(问题 7)。
- @VGalaxies 编写了 `SchemaTransactionV2`/`MetaManager` 
的大部分代码,可能了解某些会推翻本文部分内容的历史背景。
- @contrueCT 请看图数据 cache 这一侧,本提案有意把它推迟处理。
- @vaijosh 请判断 HBase 多 server 是否值得提供这个可选模式(问题 1)。

这里没有任何东西是已经定下的。如果有更简单的机制能达到同样的上界,或者这些上界对 k8s 
版本来说本身就是错的,那正是本帖希望收到的反馈。完整的分析、逐模式矩阵、失效语义和测试计划已经整理成一份更长的文档;未决问题和实施计划见下方第一条评论。

</details>


GitHub link: https://github.com/apache/hugegraph/discussions/3205

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to