GitHub user bitflicker64 edited a comment on the discussion: Cross-server
schema cache consistency without broadcast storms (#2617)
# Open questions and proposed implementation plan
Updated 2026-09-18. See the discussion timeline for merged fixes and the
separate #3157 watch-recovery PR. The phases below are proposals, not
additional scope for #3157 or an already-approved release gate.
## Open questions
Numbered so replies can reference them directly ("Q3: 5s"). Q8, Q9 and Q11
block the start of implementation; Q1 gates only the optional HBase phase; Q3
needs an answer before the release; the rest calibrate claims or clean up.
1. **Ship an HBase multi-server mode at all?** HBase is marked for removal in
2.0 (`BackendProviderFactory.java:40`). The counter-based register is a small
severable PR, but it reserves a HugeType code and adds a counter row to every
schema store. Alternative: declare HBase multi-server out of scope and document
the legacy RPC notifier as its only story. (@vaijosh, @imbajin)
2. **HugeType code reservation** for `SCHEMA_VERSION`, only if Q1 says yes: a
code near META(40)/COUNTER(50), plus a one-time sweep of serializers that
switch on type codes. Reserve an adjacent code for a possible future graph-data
epoch at the same time?
3. **Reconcile interval default:** proposed 10 s, targeting two successful
reconcile intervals after durable version publication. Confirm bounded
read/reload latency and durable recovery of a schema commit whose version write
fails before promising a deadline after every successful DDL. Acceptable for
the k8s release, or should the chart pin 5 s (halves the bound, doubles a
still-trivial poll cost)? (@imbajin)
4. **PD watch and read semantics:** are KvClient get/scan leader-routed, or can
a follower serve stale reads? Does the watch replay anything after reconnect,
and does it compact rapid puts to one key? Polling avoids reliance on watch
delivery, but correctness and timing still depend on version-read visibility
and durable publication. These guarantees need explicit confirmation. (@koi2000)
5. **Legacy event-key deprecation timeline:** new publishers dual-publish the
legacy CLEAR key for add/remove/clear bursts. One minor release of
dual-publish, or longer? Retiring it also upgrades those bursts from full-clear
to per-id granularity. Related: formally deprecate the SOFA-RPC CacheNotifier's
cache-clear role in the same cycle? (@imbajin)
6. **Out-of-tree consumers** of the dead
`notifyGraphVertexCacheClear`/`notifyGraphEdgeCacheClear` keys: in-repo grep
shows zero listeners, but the keys are visible on the wire. Does any external
tooling watch them? If provably none, the escape-hatch flag can be dropped and
the puts deleted outright.
7. **Store release cadence** for the `SchemaDriver` parser fix: deployed store
nodes benefit only via a store artifact release. When can one ship, and do
store artifacts deploy out of lockstep long-term? (@Soyaazz, @Tsukilc)
8. **Delete the `task.sync_deletion` notification gate unconditionally?** Its
premise is verifiably false across JVMs, and it also suppresses
STORE_CLEAR/TRUNCATE propagation. No config to restore it is proposed. It
changes event emission for single-JVM test setups that asserted on notification
counts. (@dpol1)
9. **Replace the V2 name-miss inline full reload with V1-style point repair**
in the same PR that ships delta application: per-id deltas make name misses
routine, and each miss currently triggers a request-thread full reload. This
changes a subtle existing recovery behavior; is anything known to depend on the
eager reload? (@dpol1, @Tsukilc)
10. **SchemaStatus audit:** the timing argument treats `status().ok()`
(CREATED) as the only safe-to-use state, verified for GraphIndexTransaction's
index gating. A pass over all SchemaStatus consumers (OLAP gates, Hubble
displays) would make the classification airtight. (@imbajin)
11. **DDL error and durable-publication contract:** today a failed event put
after a committed add fails the already-committed DDL call. The design instead
returns success, logs, raises metrics and retries in the background, because
reporting failure for a committed write misstates reality. It is an observable
contract change that clients or tests may depend on. Confirm it is acceptable
and how a committed schema change is recovered if the version write fails and
the publisher crashes before its background retry. An in-memory dirty flag
alone cannot establish the claimed convergence bound. The decided contract must
be release-noted. (@imbajin, @dpol1)
## Implementation plan
PR-sized phases, each independently landable. **The proposal recommends Phases
0 and 1 plus integration validation for the k8s release; maintainers have not
yet accepted this as a release gate**; Phase 2 is optional and severable.
**Phase 0, standalone preparatory fixes (tiny, land first):**
- PR-0a: `PdMetaDriver.extractValuesFromResponse` stops swallowing watch
batches containing non-Put events (~30 lines). Check the corresponding Struct
helper too; transport recovery in #3157 does not change this extraction
contract.
- PR-0b: `SchemaDriver` accepts both the JSON and the legacy plain-string
payload, with log-and-skip instead of `assert` (~25 lines). Fixes bug 1 from
the post.
- PR-0c: remove the dead vertex/edge cache-clear publishes behind a default-off
flag (~40 lines).
**Phase 1, proposed core machinery on the hstore+PD path (release scope pending
agreement):**
- PR-1: sync package skeleton: register SPI, coalescing flusher with guaranteed
final flush, per-graph synchronizer state, new DELTA key codec. Fully
unit-testable, no call sites yet (~550 lines).
- PR-2: reconciler singleton with per-graph due-times, graph ownership and
lifecycle (create lazily, drain on close, delete the version key on drop) (~400
lines).
- PR-3: wire the V2 transaction: register write plus coalesced events on
add/update/remove/clear, delete the `task.sync_deletion` gate, name-miss point
repair, subscribe to both event keys (~280 lines changed). Q8, Q9 and Q11 land
here.
- PR-4: consumer hardening: the generation stamp against reads that straddle a
clear, single-flight reloads (~285 lines). Should ride the same release train
as PR-3.
**Phase 2, optional HBase mode (only if Q1 says yes):**
- PR-5: counter register with self-echo accounting (~350 lines).
**Phase 3, validation and release:**
- PR-6: multi-node integration suite on the existing 3 PD + 3 store + 3 server
CI env, including a storm-regression test asserted on metrics (event ceiling,
resync cadence, zero legacy events for status-only bursts) (~400 lines).
- PR-7: docs, conf templates, release notes covering the upgrade matrix and the
Q11 contract change (~120 lines).
- Pre-release, out of repo: a helm-chart validation pass replaying the literal
#2617 Hubble scenario across 2+ replicas, a mixed-image rolling upgrade, PD
leader chaos, and a 1-hour soak. Two load figures (register-put absorption,
scan sizing) are estimates until this pass confirms them.
**Phase 4, deferred:** graph-data cache events reusing the same machinery,
orphan-key janitor, RPC notifier deprecation or repair, legacy-key retirement
per Q5.
## Next step
No implementation until Q8, Q9 and Q11 have answers and the overall direction
has a maintainer nod. Alternative designs that reach the same bounds with less
machinery are welcome; the storm numbers in the post are the bar any
alternative has to clear.
GitHub link:
https://github.com/apache/hugegraph/discussions/3205#discussioncomment-18431491
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]