GitHub user bitflicker64 added a comment to the discussion: Cross-server schema
cache consistency without broadcast storms (#2617)
# Open questions and implementation plan
## Open questions
Numbered so replies can reference them directly ("Q3: 5s"). Q8, Q9 and Q11
block the start of implementation; Q1 gates only the optional HBase phase; Q3
needs an answer before the release; the rest calibrate claims or clean up.
1. **Ship an HBase multi-server mode at all?** HBase is marked for removal in
2.0 (`BackendProviderFactory.java:40`). The counter-based register is a small
severable PR, but it reserves a HugeType code and adds a counter row to every
schema store. Alternative: declare HBase multi-server out of scope and document
the legacy RPC notifier as its only story. (@vaijosh, @imbajin)
2. **HugeType code reservation** for `SCHEMA_VERSION`, only if Q1 says yes: a
code near META(40)/COUNTER(50), plus a one-time sweep of serializers that
switch on type codes. Reserve an adjacent code for a possible future graph-data
epoch at the same time?
3. **Reconcile interval default:** proposed 10 s, giving a 20 s worst-case
staleness ceiling. Acceptable for the k8s release, or should the chart pin 5 s
(halves the bound, doubles a still-trivial poll cost)? (@imbajin)
4. **PD watch and read semantics:** are KvClient get/scan leader-routed, or can
a follower serve stale reads? Does the watch replay anything after reconnect,
and does it compact rapid puts to one key? Correctness does not depend on the
answers (the poll carries it), but they set the honest typical-latency number.
(@koi2000)
5. **Legacy event-key deprecation timeline:** new publishers dual-publish the
legacy CLEAR key for add/remove/clear bursts. One minor release of
dual-publish, or longer? Retiring it also upgrades those bursts from full-clear
to per-id granularity. Related: formally deprecate the SOFA-RPC CacheNotifier's
cache-clear role in the same cycle? (@imbajin)
6. **Out-of-tree consumers** of the dead
`notifyGraphVertexCacheClear`/`notifyGraphEdgeCacheClear` keys: in-repo grep
shows zero listeners, but the keys are visible on the wire. Does any external
tooling watch them? If provably none, the escape-hatch flag can be dropped and
the puts deleted outright.
7. **Store release cadence** for the `SchemaDriver` parser fix: deployed store
nodes benefit only via a store artifact release. When can one ship, and do
store artifacts deploy out of lockstep long-term? (@Soyaazz, @Tsukilc)
8. **Delete the `task.sync_deletion` notification gate unconditionally?** Its
premise is verifiably false across JVMs, and it also suppresses
STORE_CLEAR/TRUNCATE propagation. No config to restore it is proposed. It
changes event emission for single-JVM test setups that asserted on notification
counts. (@dpol1)
9. **Replace the V2 name-miss inline full reload with V1-style point repair**
in the same PR that ships delta application: per-id deltas make name misses
routine, and each miss currently triggers a request-thread full reload. This
changes a subtle existing recovery behavior; is anything known to depend on the
eager reload? (@dpol1, @Tsukilc)
10. **SchemaStatus audit:** the timing argument treats `status().ok()`
(CREATED) as the only safe-to-use state, verified for GraphIndexTransaction's
index gating. A pass over all SchemaStatus consumers (OLAP gates, Hubble
displays) would make the classification airtight. (@imbajin)
11. **DDL error contract change:** today a failed event put after a committed
add fails the already-committed DDL call. The design instead returns success,
logs, raises metrics and retries in the background, because reporting failure
for a committed write misstates reality. It is an observable contract change
that clients or tests may depend on. Confirm it is acceptable; it will be
release-noted either way. (@imbajin, @dpol1)
<details>
<summary>中文</summary>
编号是为了方便回复时直接引用(比如「Q3: 5s」)。Q8、Q9 和 Q11 阻塞实现的启动;Q1 只影响可选的 HBase 阶段;Q3
需要在发版之前有结论;其余的用于校准结论或做清理。
1. **到底要不要提供 HBase 多 server 模式?** HBase 已被标记为在 2.0
中移除(`BackendProviderFactory.java:40`)。基于 counter 的 register(版本寄存器)是一个可以独立切分出去的小
PR,但它会占用一个 HugeType code,并给每个 schema store 增加一行 counter。另一种做法:声明 HBase 多 server
不在范围之内,并把遗留的 RPC notifier 记录为它唯一的方案。(@vaijosh、@imbajin)
2. **为 `SCHEMA_VERSION` 预留 HugeType code**,仅当 Q1 的答案为是:选一个靠近
META(40)/COUNTER(50) 的 code,外加对所有按 type code 分支的 serializer
做一次性梳理。是否同时为未来可能出现的图数据 epoch 预留一个相邻的 code?
3. **对账间隔的默认值:** 提案为 10 s,对应 20 s 的最坏陈旧上界。这对 k8s 版本可以接受吗,还是 chart 应当固定为 5
s(上界减半,而轮询成本翻倍后依然可以忽略)?(@imbajin)
4. **PD 的 watch 与读语义:** KvClient 的 get/scan 是否路由到 leader,还是 follower
也可能返回陈旧的读结果?watch 在重连之后是否会重放任何内容,它是否会把对同一个 key 的密集 put
压缩成一个?正确性并不依赖这些答案(由轮询来兜底),但它们决定了典型延迟这个数字能否如实给出。(@koi2000)
5. **遗留事件 key 的废弃时间表:** 新的发布方会对 add/remove/clear 突发(burst)在遗留的 CLEAR key
上做双发布。双发布保留一个 minor 版本,还是更久?取消它同时还能把这些突发从全量清空升级为按 id 的粒度。相关问题:是否在同一个周期内正式废弃
SOFA-RPC CacheNotifier 承担的 cache 清空职责?(@imbajin)
6. **已失效的 `notifyGraphVertexCacheClear`/`notifyGraphEdgeCacheClear` 这两个 key
的仓库外消费者:** 仓库内 grep 的结果是零个 listener,但这些 key 在网络上是可见的。是否有外部工具在 watch
它们?如果可以证明没有,那么逃生开关可以去掉,这些 put 也可以直接删除。
7. **`SchemaDriver` 解析修复的 store 发版节奏:** 已部署的 store 节点只有通过一次 store
制品发版才能受益。什么时候可以发一版,store 制品长期来看是否会与其他组件不同步发布?(@Soyaazz、@Tsukilc)
8. **是否无条件删除 `task.sync_deletion` 这个通知开关?** 它的前提在跨 JVM 的场景下可被验证为不成立,而且它还抑制了
STORE_CLEAR/TRUNCATE 的传播。这里不打算提供任何用于恢复它的配置。它会改变单 JVM
测试环境下的事件发出行为,那些对通知次数做断言的测试会受影响。(@dpol1)
9. **在承载 delta 应用的同一个 PR 中,用 V1 风格的定点修复替换 V2 的 name 未命中内联全量 reload:** 按 id 的
delta 会让 name 未命中变成常态,而目前每一次未命中都会在请求线程上触发一次完整
reload。这改变了一个细微的既有恢复行为;是否已知有什么东西依赖这种主动的 reload?(@dpol1、@Tsukilc)
10. **SchemaStatus 梳理:** 时序上的论证把 `status().ok()`(CREATED)当作唯一可安全使用的状态,这一点已针对
GraphIndexTransaction 的 index 门控做过核对。对所有 SchemaStatus 消费方(OLAP 门控、Hubble
展示)再过一遍,可以让这个分类无懈可击。(@imbajin)
11. **DDL 错误契约的变更:** 今天,在 add 已提交之后事件 put 失败,会让这次已经提交的 DDL
调用返回失败。本设计改为返回成功、记录日志、上报指标并在后台重试,因为对一次已提交的写入报告失败是对现实的错误陈述。这是一个可观测的契约变更,客户端或测试可能依赖它。请确认这是可以接受的;无论结论如何都会写进
release note。(@imbajin、@dpol1)
</details>
## Implementation plan
PR-sized phases, each independently landable. **Phases 0 and 1 plus the
integration suite are the set needed before the k8s release**; Phase 2 is
optional and severable.
**Phase 0, standalone preparatory fixes (tiny, land first):**
- PR-0a: `PdMetaDriver.extractValuesFromResponse` stops swallowing watch
batches containing non-Put events (~30 lines).
- PR-0b: `SchemaDriver` accepts both the JSON and the legacy plain-string
payload, with log-and-skip instead of `assert` (~25 lines). Fixes bug 1 from
the post.
- PR-0c: remove the dead vertex/edge cache-clear publishes behind a default-off
flag (~40 lines).
**Phase 1, core machinery on the hstore+PD path (chart-blocking):**
- PR-1: sync package skeleton: register SPI, coalescing flusher with guaranteed
final flush, per-graph synchronizer state, new DELTA key codec. Fully
unit-testable, no call sites yet (~550 lines).
- PR-2: reconciler singleton with per-graph due-times, graph ownership and
lifecycle (create lazily, drain on close, delete the version key on drop) (~400
lines).
- PR-3: wire the V2 transaction: register write plus coalesced events on
add/update/remove/clear, delete the `task.sync_deletion` gate, name-miss point
repair, subscribe to both event keys (~280 lines changed). Q8, Q9 and Q11 land
here.
- PR-4: consumer hardening: the generation stamp against reads that straddle a
clear, single-flight reloads (~285 lines). Should ride the same release train
as PR-3.
**Phase 2, optional HBase mode (only if Q1 says yes):**
- PR-5: counter register with self-echo accounting (~350 lines).
**Phase 3, validation and release:**
- PR-6: multi-node integration suite on the existing 3 PD + 3 store + 3 server
CI env, including a storm-regression test asserted on metrics (event ceiling,
resync cadence, zero legacy events for status-only bursts) (~400 lines).
- PR-7: docs, conf templates, release notes covering the upgrade matrix and the
Q11 contract change (~120 lines).
- Pre-release, out of repo: a helm-chart validation pass replaying the literal
#2617 Hubble scenario across 2+ replicas, a mixed-image rolling upgrade, PD
leader chaos, and a 1-hour soak. Two load figures (register-put absorption,
scan sizing) are estimates until this pass confirms them.
**Phase 4, deferred:** graph-data cache events reusing the same machinery,
orphan-key janitor, RPC notifier deprecation or repair, legacy-key retirement
per Q5.
<details>
<summary>中文</summary>
按 PR 粒度划分的阶段,每个阶段都可以独立合入。**阶段 0 和阶段 1 加上集成测试套件,是 k8s 发版之前必须完成的部分**;阶段 2
是可选的,可以切分出去。
**阶段 0,独立的准备性修复(很小,先合入):**
- PR-0a:`PdMetaDriver.extractValuesFromResponse` 不再吞掉包含非 Put 事件的 watch 批次(约 30
行)。
- PR-0b:`SchemaDriver` 同时接受 JSON 和遗留的纯字符串 payload,用记日志并跳过代替 `assert`(约 25
行)。修复正文中的 bug 1。
- PR-0c:在一个默认关闭的开关后面移除已失效的 vertex/edge cache 清空发布(约 40 行)。
**阶段 1,hstore+PD 路径上的核心机制(阻塞 chart):**
- PR-1:sync 包骨架:register SPI、带保证最终刷新的合并 flusher、按 graph 的同步器状态、新的 DELTA key
编解码。完全可做单元测试,暂时没有调用点(约 550 行)。
- PR-2:对账器(reconciler)单例,带按 graph 的到期时间、graph 归属与生命周期(惰性创建,关闭时排空,删除时清除版本 key)(约
400 行)。
- PR-3:接入 V2 事务:在 add/update/remove/clear 时写 register 并发出合并后的事件,删除
`task.sync_deletion` 开关,name 未命中改为定点修复,同时订阅两个事件 key(改动约 280 行)。Q8、Q9 和 Q11
在这里落地。
- PR-4:消费端加固:用于防止读请求跨越一次清空的 generation stamp,以及 single-flight 的 reload(约 285
行)。应当与 PR-3 搭同一班发版车。
**阶段 2,可选的 HBase 模式(仅当 Q1 的答案为是):**
- PR-5:带自回声计账的 counter register(约 350 行)。
**阶段 3,验证与发版:**
- PR-6:在现有的 3 PD + 3 store + 3 server 的 CI
环境上做多节点集成测试套件,包含一个以指标做断言的风暴回归测试(事件上限、resync 节奏、纯 status 突发时零遗留事件)(约 400 行)。
- PR-7:文档、conf 模板、覆盖升级矩阵和 Q11 契约变更的 release note(约 120 行)。
- 发版前,在仓库之外:一次 helm chart 验证,在 2 个以上副本上原样复现 #2617 的 Hubble 场景,加上混合镜像的滚动升级、PD
leader 混沌测试,以及 1 小时的 soak。有两个负载数字(register put 的吸收能力、扫描规模)在这一轮验证确认之前都只是估算。
**阶段 4,推迟:** 复用同一套机制的图数据 cache 事件、孤儿 key 清理器、RPC notifier 的废弃或修复、按 Q5 结论下线遗留
key。
</details>
## Next step
No implementation until Q8, Q9 and Q11 have answers and the overall direction
has a maintainer nod. Alternative designs that reach the same bounds with less
machinery are welcome; the storm numbers in the post are the bar any
alternative has to clear.
<details>
<summary>中文</summary>
在 Q8、Q9 和 Q11 有答案、并且整体方向得到 maintainer
认可之前,不会开始实现。如果有用更少机制就能达到同样上界的替代设计,非常欢迎提出;正文中的风暴数字就是任何替代方案需要跨过的门槛。
</details>
GitHub link:
https://github.com/apache/hugegraph/discussions/3205#discussioncomment-18431491
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]