bitflicker64 opened a new pull request, #3481: URL: https://github.com/apache/hugegraph/pull/3481
## Purpose of the PR - close #3235 > Replaces #3477, which was opened from a personal fork and stopped at `7df011acb`. This PR is opened from the org branch `hugegraph/hugegraph:fix/schema-cache-events-3235`, the branch of hugegraph/hugegraph#374, so both PRs have one head. It has the three commits of #3477 and the maintainer follow-up [d2d54fe](https://github.com/hugegraph/hugegraph/commit/d2d54fe0ee1b1dec9114a4d755b3c839fc8edabc). The review threads of #3477 are answered there. <img width="1536" height="1024" alt="image" src="https://github.com/user-attachments/assets/1abcb345-92e5-493e-9670-80e6297c0232" /> With several Servers on HStore, a schema change made through one Server does not reliably reach the others, and a Server that keeps an old copy can write it back over newer schema. #3235 has the failures measured on master `8828d8238`. This PR completes the existing schema-cache-clear event. It does not add a new sync mechanism. There is no change to PD raft, the PD store or `kv.proto`; the only PD code touched is the client library (`KvClient`). It is a new change written from master. It is not built on #3237 (polling, closed) and contains nothing from #3376, which stays separate follow-up work. ### Scope at a glance | In this PR | Not in this PR, separate follow-up | |---|---| | Publish a clear event after every schema add, update, remove and store clear or truncate, also when the write fails partway | ACK or barrier between Servers before an index rebuild starts | | Clear the V2 schema caches when the PD watch starts or restarts | Schema write and event as one atomic PD commit | | Retry a publish that failed | Protection for two Servers changing the same element at once (#3376) | | Generation guard so an old read cannot refill a cleared cache | Refusing requests while cut off from PD, and a keepalive on the watch | The right column is not solved here and this PR does not claim it. Each item is explained under "What this PR does not implement". <img width="1280" height="720" alt="schema-cache1" src="https://github.com/user-attachments/assets/aa05f3fe-e402-4f78-9328-c3fb88ac6ad4" /> Next <img width="1280" height="720" alt="schema-cache2" src="https://github.com/user-attachments/assets/7d350fa8-997d-4924-ae2f-ff1fd1b4cbd5" /> ## Main Changes ### What this PR fixes All changes are in `CachedSchemaTransactionV2` unless noted. 1. **Schema update and removal propagation.** `updateSchema()` published nothing, so a property append, `user_data`, the index list of a base label and every status change stayed on the writer. `removeSchema()` and store clear or truncate published only with `task.sync_deletion=false`. Now every committed add, update, remove and store clear or truncate publishes a clear event after its write. Every event clears at once; events are not delayed or merged. A schema write is two or three PD puts. If it fails after the first one, PD holds part of the change. That case publishes too, and the writer drops its own copy of the element; the caller still gets the error. 2. **Cache invalidation on PD watch start and recovery.** Events are delivered at most once, and `KvClient` subscribes again one second after a disconnect without telling the listener. `KvClient.listen()` has a new overload with a callback that runs each time a watch goes live (the first `Starting` or `Started` frame confirming registration on a new stream, including after a leader change or a stream error; an `Alive` heartbeat alone does not confirm registration). The Server clears all its V2 schema caches when it runs, so nothing missed while the watch was down survives. 3. **Failed-publication retry.** A graph whose publish failed goes into an in-memory pending set and is published again after 1 s, doubling up to 30 s, and at once when the watch is live again. Each Server also publishes once when it opens a graph, which repairs a publish lost to a restart of the writer. 4. **Generation guard.** A generation counter and a read-write lock on the shared cache. Every clear, removal and local write advances the generation as the last step of its change. A read by id, a read by name, a full listing or the writer's own write-through that started before a clear cannot cache its value after the clear. A name miss now triggers one reload per graph at a time instead of one per waiting thread. Plumbing for the callback: `MetaDriver.listen(key, consumer, onStarted)` as a default method that falls back to the plain listen, overridden in `PdMetaDriver`, and passed through `MetaManager` and `GraphMetaManager`. The etcd meta driver uses the fallback and gets no watch-start clear. ### Changed behavior of schema calls Once the schema write has committed, a failed event publication no longer fails the schema call. The call returns success and the publication is retried in the background (item 3). Before this PR the add path threw after the commit when the publish failed, so the caller saw an error for a change that was already stored. Inside an index rebuild job the same throw would fail the job after its status write had committed. A publish failure is logged at WARN with the graph name. What a caller can now observe: the call succeeds while other Servers still hold the old copy until the retry goes through or their watch restarts. ### How this relates to the review of #3237 That review asked for a notification design in place of polling and listed points to settle first. This is where each one stands here. Nothing from #3237 itself is revived. | Point from that review | In this PR | |---|---| | Notify, do not poll | Done. Nothing runs while the schema is idle. A retry timer exists only while a publish is pending. | | Restore update and status notifications | Done, item 1. This reverses the choice in #3011 to keep status changes local. The cost is measured under "Cost of an event". | | Retry without blocking DDL | Done, item 3. | | A reliable link between a schema commit and its notification | Partly. A failed publish is retried, a restarted writer publishes again, and a write that fails partway publishes too. The write and the event are still separate PD puts: see the limits. | | A Server that was away reloads before it serves | Partly. It drops all its cached schema when its watch is live again. It is not kept from serving while it is cut off: see the limits. | | Batch redundant invalidations within one operation | Not done. One index creation publishes four events. Merging them delays the clear of the base label, which widens the index window described in the limits. | ### Cost of an event One event makes every other Server drop the schema cache of that graph. Its next read by name then loads the whole schema again (four prefix scans in PD). Measured on 1 PD + 1 Store + 3 Servers on one host (12 CPUs), fresh cluster with 300 property keys, 2026-11-13: 200 updates in a row through S0 while S1 and S2 each serve schema reads by name on two threads. Two runs per build. | | master `8828d8238` | this PR | |---|---|---| | One update on S0, mean | 7.8 and 8.0 ms | 11.8 and 11.8 ms | | Reads per second on S1 and S2 while idle | 3,900 and 4,140 | 3,850 and 4,310 | | Reads per second during the burst | 3,610 and 3,760 | 320 and 340 | | Median read during the burst | 1.0 and 1.1 ms | 10.6 and 10.8 ms | | Reads per second right after the burst | 4,780 and 4,890 | 4,950 and 4,970 | | Failed reads | 0 | 0 | | S1 and S2 show the last update 3 s after the burst | no | yes | So for as long as schema changes arrive back to back, schema reads on the other Servers were about 11 times slower in this setup, and they recovered when the burst ended. The cost grows with the schema: the same burst on a graph with about 950 property keys gave 180 to 190 reads per second. Ten index creations in a row cost about the same on both builds (140 to 149 ms per creation and 2,450 to 2,580 reads per second on the other Servers). I did not measure the effect on graph reads and writes, only on schema reads. Lowering this cost means sending which element changed, or merging events, and is proposed as follow-up. ### What this PR does not implement These are known limits. They are listed in `docs/server-runtime.md` as limits, and none of them is implied to be solved by this PR. Follow-up issues so far: the existing #3376, and #3478 for a watch that stops for good. | Not implemented | What can still go wrong | Why it is excluded | |---|---|---| | ACK or barrier before an index rebuild | A write on another Server that is prepared from the old base label and committed after the rebuild scan has passed it is missing from the new index. `GraphTransaction.prepareCommit()` takes the index list from the cached label, and `IndexLabelRebuildJob` scans the label once and starts as soon as the task is scheduled. | The rebuild would have to wait until every Server has confirmed the new base label. That needs Server membership, confirmations and a rule for a Server that never answers. It is a protocol of its own and belongs in a separate design. | | Atomic schema commit plus event publication | A writer that stops for good between its schema write and its publish leaves the other Servers on the old copy until the next event for that graph or their next watch restart. If the writer restarts, its publish on graph open repairs this. | The write and the event are two PD puts. Making them one needs a PD-side change (one raft entry for both, or an outbox), which this PR avoids on purpose. | | Concurrent same-element schema changes | An update still writes the whole element from the writer's copy. Two Servers changing the same element at the same moment can lose one change. | It needs a conditional write in PD. That is #3376, which is +3,711 lines on its own. | | Offline-serving gate | A Server that cannot reach PD keeps serving its cached schema until it reconnects. | Refusing requests in that state trades availability for freshness. That is a product decision, not part of a cache fix. | | Restart of a watch that stopped for good | A PD watch that ends with a gRPC status the PD client does not retry (`UNAUTHENTICATED`, `PERMISSION_DENIED`, `UNIMPLEMENTED` and six more) is removed and not started again. The watch-start clear then never runs, and that Server keeps its cached schema until it is restarted; its log has one ERROR line for the stopped watch. PD does not answer a watch with these codes itself. They can come from a proxy between Server and PD: gRPC maps HTTP 401, 403 and 404 to three of them. | This is how the PD client on master behaves for every Server meta listener, not only this one. Starting the watch again needs a stop callback through the meta driver and a limit on how often a refused watch is retried. Tracked in #3478. | | Watch keepalive | A watch stream that dies without an error is noticed only when another PD call from that Server fails. | It needs keepalive settings on the PD client channels, which affects every user of the PD client and needs its own tests. | Two more points: - A reload right after a PD leader change also needs #3383, so the read cannot be served by a leader that has not applied its log yet. - The guarantees start when the last Server runs this version. A Server on the old version still publishes nothing on update. Index entries skipped before the upgrade stay missing until that index is rebuilt. Found while reviewing this change and left out because they are independent of it: - `IntObjectMap.set()` grows its array by one doubling only, so the first write of a schema id above twice the current size throws. It is on master and is hit by the array cache of both schema transactions. It can no longer skip a publish, because the publish runs in a `finally`. - Store nodes listen on the same clear key (`SchemaDriver`) without a watch-start callback. I did not examine what their cache is used for. ### Review scope This PR is meant to stay at its current size and shape. For a review finding I will do the following: - A bug in this patch, or something that breaks one of the four items under "What this PR fixes": fixed here, with the smallest change and a regression test. - Anything that needs one of the excluded items above (ACK or barrier, atomic commit, conditional write, offline gate, keepalive), a PD server change, or a refactor of code this PR does not touch: answered with the reason and proposed as a follow-up issue, not added here. The second commit is the result of such a pass over the first one. It fixes four things, each with a test: no event after a schema write that failed partway, a retry that could schedule itself twice, a generation that advanced before the caches had changed (a concurrent read could copy the old value into the array cache), and a listing flag that never recovered after a refused write-through. The third commit follows the review comments on the mirror, hugegraph/hugegraph#374. It changes no production code. The index test now fails when one of its background readers stops on an error (their results were never read, so the test could pass with fewer reads than it claims), and the stopped-watch limit above is added to `docs/server-runtime.md` with #3478 as its follow-up. The remaining comments were checked against the code and not taken: the `Response` in the reader loop is released by `readEntity`, the `Scanner` reads a stream that its `try` closes, a stopped watch is already logged at ERROR by the PD client, a null `onStarted` is already rejected by name in the PD client, and the shared test helper was retained. HTTP timeouts were initially deferred; the maintainer follow-up now sets finite timeouts on this test class's targets and removes the TODO. A second round of comments on the third commit initially led to no code change. Two points were new: isolation from an already-running retry, and the retry thread remaining after the pending set is empty. The maintainer follow-up fixes the isolation issue by invalidating the retry epoch and draining the executor outside the monitor before replacing the test publisher, with a deterministic regression test. The executor remains one idle daemon thread, created only after a first failed publish, consistent with other Server-core schedulers. ### Size before maintainer follow-up 16 files, +2,692 / -121 (`git diff --shortstat 8828d8238 7df011acb`). | Part | Added | Removed | |---|---|---| | Production code (`CachedSchemaTransactionV2`, `KvClient`, meta plumbing) | 628 | 111 | | Tests | 1,946 | 3 | | Cluster-test harness (restart one PD node in place) | 49 | 2 | | Docs | 69 | 5 | ## Verifying these changes - [ ] Trivial rework / code cleanup without any test coverage. (No Need) - [ ] Already covered by existing tests, such as *(please modify tests here)*. - [x] Need tests and can be verified as follows: - `CachedSchemaTransactionV2Test` (new, 36 tests): publish on every write path and after the write, failed and partial writes, failed publish and retry, watch start, and the generation guard on every cache insertion and write-through path - `KvClientTest` (10 new tests): the watch-start callback on first start, leader change, stream error and a replaced watch, including an `Alive` heartbeat arriving before registration confirmation - `MultiClusterSchemaCacheTest` (new, 5 tests): 3 PD, 3 Store and 3 Servers ### Reproductions before and after Same scripts for both columns, through the REST API, 1 PD + 1 Store + 3 Servers (S0, S1, S2) as local processes on one host (12 CPUs, Java 17), 2026-11-13. "Before" is master `8828d8238`, "after" is the pre-maintainer PR revision used for these 2026-11-13 measurements. | Case | Before | After | |---|---|---| | Update `user_data` through S0 | Still missing on S1 and S2 after 30 s | Visible on S1 and S2 at the first check, after 1 s | | S0 appends `p1`, then S1 appends `p2` to the same label | PD ends with `[a, p2]`; a vertex write that sets `p1` returns 400 on S2 | All three Servers show `[a, p1, p2]`; the vertex is read back with `p1` through each Server | | Index status, 30 index creations through S0 | S1 and S2 keep `CREATING` in 30 of 30 | 0 of 30 | | Base label, same 30 runs | Index missing from the label on S1 in 14 of 30 and on S2 in 15 of 30; 29 of 29 vertices written through such a Server are not indexed | 0 of 30; 0 vertices not indexed | | Add through S0 right after a PD restart | Missing on S1 and S2 after 60 s, 3 of 3 runs | Visible within 2 s, 3 of 3 runs | | Remove with `task.sync_deletion=true` | Still listed on S1 and S2 after 30 s | Gone on all Servers within 1 s | In the "after" run each Server logged the watch-start clear four times: once at start and once per PD restart. No Server logged a failed publish. All of this ran on one host with sub-millisecond latency. It shows that the cases happen and stop happening, not how often they occur in production. ### Author test suites before maintainer follow-up Java 17, 2026-11-13. Format and multi cluster ran on the tree of the head commit. The other suites ran on the second commit; the third changes one cluster test and one doc file and adds no file. Server unit and HStore core ran before one doc paragraph and one comment in the cluster-test harness were added to the second commit. | Suite | Command | Result | |---|---|---| | Format | `mvn editorconfig:check -pl hugegraph-pd/hg-pd-test,hugegraph-server/hugegraph-test,hugegraph-cluster-test/hugegraph-clustertest-test -am -ntp` | BUILD SUCCESS | | Licence headers | `mvn apache-rat:check -ntp` | BUILD SUCCESS | | Server unit | `mvn test -pl hugegraph-server/hugegraph-test -am -P unit-test -ntp` | 1,477 run, 0 failures, 8 skipped | | HStore core (PD and Store running) | `mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,hstore -ntp` | 1,033 run, 0 failures, 77 skipped | | Multi cluster | `mvn test -pl hugegraph-cluster-test/hugegraph-clustertest-test -am -P multi-cluster-test -ntp` | 11 run, 0 failures (5 are `MultiClusterSchemaCacheTest`); the runtime log check passes. Run again on the third commit with the same result | | PD client (PD running) | `mvn test -pl hugegraph-pd/hg-pd-test -am -P pd-client-test -ntp` | 122 run, 0 failures (`KvClientTest`: 48), run on the first commit; historical evidence, superseded for the modified PD client by the follow-up validation below | The two cache test classes were also run 12 times in a row in fresh JVMs: 12 of 12 passed. One existing test changed: `CachedSchemaTransactionTest.testV2WarmRequestsRetainSchemaWithoutMetadataReadsAndReceiveInvalidations` expected one PD put and now expects one at graph open and one more for the store clear. ### Do the tests detect the bugs - `MultiClusterSchemaCacheTest` on unpatched master `8828d8238`: 4 of 5 fail (update, append, index creation, PD restart). - Each fix was taken out in turn and `CachedSchemaTransactionV2Test` was run against it: | Change taken out | Tests that fail | |---|---| | Publish after the write (publish moved before it) | `testEveryPublishComesAfterItsStoreWrite`, `testAddPublishes` | | Handling of a write that failed partway | `testWriteThatFailsHalfwayPublishesAndDropsTheElement`, `testRemoveThatFailsHalfwayPublishesAndDropsTheElement`, `testFailedWriteKeepsItsEventWhenThePublishFailsToo` | | Only the newest retry reschedules | `testWatchStartDuringARetryLeavesOneRetry` | | Generation advanced last in a local write | `testReadDuringALocalWriteDoesNotPutTheOldValueBack` (3 of 3 runs) | | Eviction after a refused write-through | `testRefusedWriteThroughEvictsWhatWasCachedMeanwhile`, `testReadsNeverGoBackBehindAWriteUnderConcurrentClears` | | Read lock around a cached listing | `testAClearWaitsForAListingReadFromTheCache`, `testListingIsNeverPartialWhileClearsRun`, `testReadsNeverGoBackBehindAWriteUnderConcurrentClears` | | Whole-list flag set after a full load | `testRefusedWriteThroughEvictsWhatWasCachedMeanwhile` | ### Known gaps in the evidence - `testRemoveReachesEveryServerWithSyncDeletion` passes on unpatched master, because a new graph publishes events while its system schema is set up. The unit test and the manual check in the table above cover that gate. - `testChangeDuringPDRestartReachesEveryServer` does not prove that a watch was down when the change committed. It failed on unpatched master in the one control run. The evidence for the watch-start clear is the unit tests plus the Server log lines of the manual run. - `testReadsNeverGoBackBehindAWriteUnderConcurrentClears` and `testListingIsNeverPartialWhileClearsRun` depend on timing. The first one failed in about half of the runs before the generation was advanced last, and in none of 12 after. - The generation order is tested with a deterministic test for a local write. For a clear it is covered only by the timing test above. - The publish in `finally` after a failure inside the cache update has no test. - The in-flight write window during index creation and the writer that never returns are derived from the code and were not reproduced. - Not tested: a real network partition, a silent stream death, Servers on mixed versions. - Not rerun locally for the maintainer follow-up: api-test, the TinkerPop suites, the PD core, common and REST suites, the Store suites and the simple-cluster suite. Historical CI and current-head validation are recorded separately below. ### Maintainer follow-up and validation: `d2d54fe` Reviewed from three independent angles (cache correctness, watch/retry state transitions, and existing comments/test quality), followed by adversarial cross-checks. The follow-up changes 5 files, +156 / -15 relative to `7df011acb`; only +15 / -7 are in production code. It adds no PD-server/proto change, polling, ACK/barrier, outbox, or extra retry abstraction. - Separate receipt of a transport frame from confirmed watch registration; run the started callback only on the first `Starting` / `Started` of the current attempt. - Assert HTTP 2xx in background readers, and set this cluster test's targets to a 5 s connect timeout and 15 s read timeout. - Invalidate the retry epoch and wait for in-flight test publication to finish before replacing the test publisher. - Document that DDL for the same schema element must be serialized through one pinned Server; verify visibility before changing writers. Remove the unconditional no-request-failure claim. Validation on the exact tree pushed as `d2d54fe`, Java 17, 2026-11-15 (Asia/Singapore): | Validation | Result | |---|---| | Full 38-module clean compile, packaging, editorconfig format/check, and diff whitespace check | Passed | | `CachedSchemaTransactionV2Test` + `CachedSchemaTransactionTest` | 36 + 21 = **57 passed** | | Full `KvClientTest` against an actual PD service | **51 passed** | | `MultiClusterSuiteTest`, actual 3 PD + 3 Store + 3 Server processes | **11 passed**, including all 5 schema-cache tests; runtime log check passed | | Total distinct positive tests in this follow-up | **119 passed; 0 failures, 0 errors, 0 skipped** | | Restore old watch-start logic while retaining new regressions | All **3** new tests fail at their intended assertions | | Restore old reset helper while retaining the in-flight regression | The **1** new test fails at its intended assertion | The fixed code was restored and its tests passed after both mutation checks. Cache tests used Mockito's supported explicit startup Java agent because self-attachment is unavailable in this environment. Multi-node child JVMs used a 512 MiB heap cap, 2 active processors, and 128 MiB direct-memory/RocksDB limits; the test configuration uses Store replica count 1. These are functional checks, not a performance benchmark or evidence for three-replica behavior. The author's earlier throughput measurements were not rerun. The successful [ASF workflow run 43432139802](https://github.com/apache/hugegraph/actions/runs/43432139802) applies to `7df011acb`, the head of #3477. The CI of this PR is the first ASF run on `d2d54fe`. The permanent stopped-watch recovery gap remains tracked by #3478. Strong concurrent DDL protection, an index-rebuild barrier, atomic schema/event publication, and offline-serving policy remain separate design work. ## Does this PR potentially affect the following parts? - [ ] Dependencies ([add/update license](https://hugegraph.apache.org/docs/contribution-guidelines/contribute/#321-check-licenses) info & [regenerate_known_dependencies.sh](../install-dist/scripts/dependency/regenerate_known_dependencies.sh)) - [ ] Modify configurations - [ ] The public API - [x] Other affects: a schema call no longer fails when only the event publication failed after the commit (see "Changed behavior of schema calls"). During a burst of schema changes, schema reads on the other Servers slow down (see "Cost of an event"). New Java methods: a `KvClient.listen()` overload with a started callback and a default method `MetaDriver.listen(key, consumer, onStarted)`. Each Server publishes one clear event per graph when it opens the graph. - [ ] Nope ## Documentation Status - [ ] `Doc - TODO`: required documentation is pending; complete it before merging. - [x] `Doc - Done`: documentation is included here or linked below. - [ ] `Doc - No Need`: no user-visible documentation is affected. Documentation files in this PR or paired hugegraph-doc PR: - `docs/server-runtime.md`: new section "Schema caches across HStore Servers" with the guarantees and the limits - `helm/hugegraph/docs/operations.md`: the paragraph on index status across Pods -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
