bitflicker64 opened a new pull request, #3237: URL: https://github.com/apache/hugegraph/pull/3237
## Purpose of the PR - Fixes #3235 With several Servers on HStore, a schema change made through one Server can stay invisible on the others indefinitely. Updates and status flips publish no PD event, on purpose, because one event per `updateSchemaStatus()` call is a broadcast storm. Events that are published can be lost, because the PD watch starts from "now" after a reconnect (#3157). This PR makes every Server converge within a bounded time without adding any event. The design and its open questions are in discussion [#3205](https://github.com/apache/hugegraph/discussions/3205). The questions had no maintainer answer yet, so this PR uses conservative defaults. Each one is listed in #3235 with how to override it. ## Main Changes  - `CachedSchemaTransactionV2` writes a new opaque version of the graph to PD after every committed schema change: add, update, status flip, remove, clear, store clear or truncate. The key is `HUGEGRAPH/{cluster}/SCHEMA_VERSION/{graphspace}/{graph}`. It sits outside `GRAPHSPACE/{gs}/{graph}/SCHEMA` because `clearAllSchema()` deletes that prefix. - New `SchemaVersionReconciler`: one task per open graph on one daemon thread per JVM. Every `schema.sync.reconcile_interval` seconds it reads the version. If the version changed, it clears that graph's schema cache and adopts the value it read before the clear. Versions are only compared for equality. - A failed version write never fails the DDL call. It logs a WARN and is retried on the next tick. Existing events are not changed: adds still publish one event per operation, and a failed event put still fails the add as it does today. - Cache guard: a clear caused by a remote change and a reader's cache update are now exclusive on the graph's `SchemaCaches`. A reader skips its update if such a clear ran after its storage read, so data read before the clear is never cached after it. Cache hit paths are unchanged, except that an id-cache hit takes the lock once when it promotes the element to the array cache. - `StandardHugeGraph.close()` stops the graph's reconciler, and `drop()` deletes the version key. - New per-graph options: `schema.sync.enabled` (default `true`) and `schema.sync.reconcile_interval` (default `10`, range 0 to 3600 seconds, 0 turns the checks off). With `schema.sync.enabled=false` there is no PD traffic and no reconciler. The cache guard above stays on either way. Bounds, from the measurements below: - A change becomes visible on every other Server within one interval plus one PD read after the version is written, whatever happens to the watch. - After a PD outage, the interval counts from when that Server's own PD client can read again. - A burst of M changes lasting D seconds costs at most `ceil(D / T) + 1` clears per Server, so cluster-wide clears no longer grow with M. Notifying on every update costs `M x (N - 1)`. ## Verifying these changes - [x] Need tests and can be verified as follows: - `SchemaVersionReconcilerTest` (16 tests) covers the reconciler with an in-memory store: - clears once per change, whatever the number of writes in between; - adopts the version read before the clear; - does not skip its own writes; - retries a failed write, including a failure during the retry; - survives read failures; - stops for a closed graph; - is re-armed after an Error. - `CachedSchemaTransactionTest` has 8 new tests: - the real `getSchema(type, id)` and `getAllSchema` paths don't cache data across a remote clear injected during the read; - a local clear doesn't change the generation; - registry start and stop, including a closed graph; - the two options; - the PD key. - `mvn test -pl hugegraph-server/hugegraph-test -am -P unit-test` on Java 11.0.32 at `e2701127`: 772 run, 0 failures, 1 skipped. Master `83ef9f3fa` has 748 run, 0 failures, 1 skipped. `mvn editorconfig:check` is clean on the changed modules. - Live, 2026-09-24, on kind: 1 PD, 1 Store, 3 Servers, installed with the chart from hugegraph/hugegraph#221, probing each Server pod directly. Before each update test every schema cache was warmed on all Servers and left for 22 s, so the name-miss reload triggered by the task scheduler could not refresh a cache by accident. - "Naive" is master plus `notifySchemaCacheClear()` in `updateSchema()`, and an INFO line per remote clear so the clears can be counted. | Change through server-0, seen on server-1 / server-2 | master `83ef9f3fa` | naive | this PR `e2701127` | |---|---|---|---| | add a property key (event published) | 0.05 s / 0.06 s | 0.05 s / 0.07 s | 0.06 s / 0.08 s | | append `user_data` to it (no event) | still old after 60 s | 0.04 s / 0.06 s | 5.32 s / 5.97 s | | append again right after the PD pod restarts | still old after 60 s | 0.03 s / 0.06 s | 9.30 s / 14.40 s | | full cache clears per Server during 60 s of updates, counted over 90 s | 0 (M = 5,860) | 4,980 on each remote (M = 4,980) | 7 on each Server (M = 5,000) | The 14.40 s is one skipped tick. Server-2's own PD client had not reconnected yet when its tick ran, about 3 s after the write landed through server-0. The next tick picked the change up. From that Server's log: ``` 11:49:07 [WARN] SchemaVersionReconciler - PD unreachable, schema version reconcile skipped for graph 'DEFAULT-hugegraph': ... Failed to get 'HUGEGRAPH/... 11:49:17 [INFO] SchemaVersionReconciler - PD reachable again, schema version reconcile resumed for graph 'DEFAULT-hugegraph' 11:49:17 [INFO] SchemaVersionReconciler - Schema cache of graph 'DEFAULT-hugegraph' cleared by version reconciler (...-3 -> 1790250543...) ``` Known limits, also in #3235: - If the JVM that made a change dies before its failed version write is retried, that change stays unsignalled until the next schema change of the graph. - All graphs of a JVM share one reconciler thread. While PD is unreachable, a slow read delays the other graphs' ticks. - A reconciler task killed by an Error is re-armed only when a new schema transaction of the graph is created. - The options should match on every Server. A Server with `schema.sync.enabled=false` writes no version, so its updates are not seen by the others. ## Does this PR potentially affect the following parts? - [ ] Dependencies ([add/update license](https://hugegraph.apache.org/docs/contribution-guidelines/contribute/#321-check-licenses) info & [regenerate_known_dependencies.sh](../install-dist/scripts/dependency/regenerate_known_dependencies.sh)) - [x] Modify configurations - [ ] The public API - [x] Other affects: a new PD key per graph, one extra PD put per schema change, and one PD get per graph per interval on each Server - [ ] Nope ## Documentation Status - [x] `Doc - TODO`: required documentation is pending; complete it before merging. - [ ] `Doc - Done`: documentation is included here or linked below. - [ ] `Doc - No Need`: no user-visible documentation is affected. Documentation files in this PR or paired hugegraph-doc PR: - In this PR: `hugegraph-server/hugegraph-dist/src/assembly/static/conf/graphs/hugegraph.properties` (commented examples). - Pending: two rows for the config option page in hugegraph-doc, once the option names are confirmed here: | config option | default value | description | |---|---|---| | schema.sync.enabled | true | Whether to write a per-graph schema version to PD on every schema change and check it periodically, so that a schema change made through another server clears this server's schema cache. Only for the hstore backend. Set false to keep the previous behavior: no version writes, no checks. | | schema.sync.reconcile_interval | 10 | The interval in seconds to check the schema version of a graph in PD. A schema change made through another server is visible on this server within about this interval. 0 means never check: the version is still written for other servers, and a failed write is retried by the next schema change. | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
