GitHub user bitflicker64 edited a comment on the discussion: Cross-server 
schema cache consistency without broadcast storms (#2617)

# Open questions and proposed implementation plan

Updated 2026-09-18. See the discussion timeline for merged fixes and the 
separate #3157 watch-recovery PR. The phases below are proposals, not 
additional scope for #3157 or an already-approved release gate.

## Open questions

Numbered so replies can reference them directly ("Q3: 5s"). Q8, Q9 and Q11 
block the start of implementation; Q1 gates only the optional HBase phase; Q3 
needs an answer before the release; the rest calibrate claims or clean up.

1. **Ship an HBase multi-server mode at all?** HBase is marked for removal in 
2.0 (`BackendProviderFactory.java:40`). The counter-based register is a small 
severable PR, but it reserves a HugeType code and adds a counter row to every 
schema store. Alternative: declare HBase multi-server out of scope and document 
the legacy RPC notifier as its only story. (@vaijosh, @imbajin)
2. **HugeType code reservation** for `SCHEMA_VERSION`, only if Q1 says yes: a 
code near META(40)/COUNTER(50), plus a one-time sweep of serializers that 
switch on type codes. Reserve an adjacent code for a possible future graph-data 
epoch at the same time?
3. **Reconcile interval default:** proposed 10 s, targeting two successful 
reconcile intervals after durable version publication. Confirm bounded 
read/reload latency and durable recovery of a schema commit whose version write 
fails before promising a deadline after every successful DDL. Acceptable for 
the k8s release, or should the chart pin 5 s (halves the bound, doubles a 
still-trivial poll cost)? (@imbajin)
4. **PD watch and read semantics:** are KvClient get/scan leader-routed, or can 
a follower serve stale reads? Does the watch replay anything after reconnect, 
and does it compact rapid puts to one key? Polling avoids reliance on watch 
delivery, but correctness and timing still depend on version-read visibility 
and durable publication. These guarantees need explicit confirmation. (@koi2000)
5. **Legacy event-key deprecation timeline:** new publishers dual-publish the 
legacy CLEAR key for add/remove/clear bursts. One minor release of 
dual-publish, or longer? Retiring it also upgrades those bursts from full-clear 
to per-id granularity. Related: formally deprecate the SOFA-RPC CacheNotifier's 
cache-clear role in the same cycle? (@imbajin)
6. **Out-of-tree consumers** of the dead 
`notifyGraphVertexCacheClear`/`notifyGraphEdgeCacheClear` keys: in-repo grep 
shows zero listeners, but the keys are visible on the wire. Does any external 
tooling watch them? If provably none, the escape-hatch flag can be dropped and 
the puts deleted outright.
7. **Store release cadence** for the `SchemaDriver` parser fix: deployed store 
nodes benefit only via a store artifact release. When can one ship, and do 
store artifacts deploy out of lockstep long-term? (@Soyaazz, @Tsukilc)
8. **Delete the `task.sync_deletion` notification gate unconditionally?** Its 
premise is verifiably false across JVMs, and it also suppresses 
STORE_CLEAR/TRUNCATE propagation. No config to restore it is proposed. It 
changes event emission for single-JVM test setups that asserted on notification 
counts. (@dpol1)
9. **Replace the V2 name-miss inline full reload with V1-style point repair** 
in the same PR that ships delta application: per-id deltas make name misses 
routine, and each miss currently triggers a request-thread full reload. This 
changes a subtle existing recovery behavior; is anything known to depend on the 
eager reload? (@dpol1, @Tsukilc)
10. **SchemaStatus audit:** the timing argument treats `status().ok()` 
(CREATED) as the only safe-to-use state, verified for GraphIndexTransaction's 
index gating. A pass over all SchemaStatus consumers (OLAP gates, Hubble 
displays) would make the classification airtight. (@imbajin)
11. **DDL error and durable-publication contract:** today a failed event put 
after a committed add fails the already-committed DDL call. The design instead 
returns success, logs, raises metrics and retries in the background, because 
reporting failure for a committed write misstates reality. It is an observable 
contract change that clients or tests may depend on. Confirm it is acceptable 
and how a committed schema change is recovered if the version write fails and 
the publisher crashes before its background retry. An in-memory dirty flag 
alone cannot establish the claimed convergence bound. The decided contract must 
be release-noted. (@imbajin, @dpol1)

## Implementation plan

PR-sized phases, each independently landable. **The proposal recommends Phases 
0 and 1 plus integration validation for the k8s release; maintainers have not 
yet accepted this as a release gate**; Phase 2 is optional and severable.

**Phase 0, standalone preparatory fixes (tiny, land first):**
- PR-0a: `PdMetaDriver.extractValuesFromResponse` stops swallowing watch 
batches containing non-Put events (~30 lines). Check the corresponding Struct 
helper too; transport recovery in #3157 does not change this extraction 
contract.
- PR-0b: `SchemaDriver` accepts both the JSON and the legacy plain-string 
payload, with log-and-skip instead of `assert` (~25 lines). Fixes bug 1 from 
the post.
- PR-0c: remove the dead vertex/edge cache-clear publishes behind a default-off 
flag (~40 lines).

**Phase 1, proposed core machinery on the hstore+PD path (release scope pending 
agreement):**
- PR-1: sync package skeleton: register SPI, coalescing flusher with guaranteed 
final flush, per-graph synchronizer state, new DELTA key codec. Fully 
unit-testable, no call sites yet (~550 lines).
- PR-2: reconciler singleton with per-graph due-times, graph ownership and 
lifecycle (create lazily, drain on close, delete the version key on drop) (~400 
lines).
- PR-3: wire the V2 transaction: register write plus coalesced events on 
add/update/remove/clear, delete the `task.sync_deletion` gate, name-miss point 
repair, subscribe to both event keys (~280 lines changed). Q8, Q9 and Q11 land 
here.
- PR-4: consumer hardening: the generation stamp against reads that straddle a 
clear, single-flight reloads (~285 lines). Should ride the same release train 
as PR-3.

**Phase 2, optional HBase mode (only if Q1 says yes):**
- PR-5: counter register with self-echo accounting (~350 lines).

**Phase 3, validation and release:**
- PR-6: multi-node integration suite on the existing 3 PD + 3 store + 3 server 
CI env, including a storm-regression test asserted on metrics (event ceiling, 
resync cadence, zero legacy events for status-only bursts) (~400 lines).
- PR-7: docs, conf templates, release notes covering the upgrade matrix and the 
Q11 contract change (~120 lines).
- Pre-release, out of repo: a helm-chart validation pass replaying the literal 
#2617 Hubble scenario across 2+ replicas, a mixed-image rolling upgrade, PD 
leader chaos, and a 1-hour soak. Two load figures (register-put absorption, 
scan sizing) are estimates until this pass confirms them.

**Phase 4, deferred:** graph-data cache events reusing the same machinery, 
orphan-key janitor, RPC notifier deprecation or repair, legacy-key retirement 
per Q5.

## Next step

No implementation until Q8, Q9 and Q11 have answers and the overall direction 
has a maintainer nod. Alternative designs that reach the same bounds with less 
machinery are welcome; the storm numbers in the post are the bar any 
alternative has to clear.


GitHub link: 
https://github.com/apache/hugegraph/discussions/3205#discussioncomment-18431491

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to