bitflicker64 opened a new pull request, #3481:
URL: https://github.com/apache/hugegraph/pull/3481

   ## Purpose of the PR
   
   - close #3235
   
   > Replaces #3477, which was opened from a personal fork and stopped at 
`7df011acb`. This PR is opened from the org branch 
`hugegraph/hugegraph:fix/schema-cache-events-3235`, the branch of 
hugegraph/hugegraph#374, so both PRs have one head. It has the three commits of 
#3477 and the maintainer follow-up 
[d2d54fe](https://github.com/hugegraph/hugegraph/commit/d2d54fe0ee1b1dec9114a4d755b3c839fc8edabc).
 The review threads of #3477 are answered there.
   
   <img width="1536" height="1024" alt="image" 
src="https://github.com/user-attachments/assets/1abcb345-92e5-493e-9670-80e6297c0232";
 />
   
   With several Servers on HStore, a schema change made through one Server does 
not reliably reach the others, and a Server that keeps an old copy can write it 
back over newer schema. #3235 has the failures measured on master `8828d8238`.
   
   This PR completes the existing schema-cache-clear event. It does not add a 
new sync mechanism. There is no change to PD raft, the PD store or `kv.proto`; 
the only PD code touched is the client library (`KvClient`).
   
   It is a new change written from master. It is not built on #3237 (polling, 
closed) and contains nothing from #3376, which stays separate follow-up work.
   
   ### Scope at a glance
   
   | In this PR | Not in this PR, separate follow-up |
   |---|---|
   | Publish a clear event after every schema add, update, remove and store 
clear or truncate, also when the write fails partway | ACK or barrier between 
Servers before an index rebuild starts |
   | Clear the V2 schema caches when the PD watch starts or restarts | Schema 
write and event as one atomic PD commit |
   | Retry a publish that failed | Protection for two Servers changing the same 
element at once (#3376) |
   | Generation guard so an old read cannot refill a cleared cache | Refusing 
requests while cut off from PD, and a keepalive on the watch |
   
   The right column is not solved here and this PR does not claim it. Each item 
is explained under "What this PR does not implement".
   
   <img width="1280" height="720" alt="schema-cache1" 
src="https://github.com/user-attachments/assets/aa05f3fe-e402-4f78-9328-c3fb88ac6ad4";
 />
   
   Next
   
   <img width="1280" height="720" alt="schema-cache2" 
src="https://github.com/user-attachments/assets/7d350fa8-997d-4924-ae2f-ff1fd1b4cbd5";
 />
   
   ## Main Changes
   
   ### What this PR fixes
   
   All changes are in `CachedSchemaTransactionV2` unless noted.
   
   1. **Schema update and removal propagation.** `updateSchema()` published 
nothing, so a property append, `user_data`, the index list of a base label and 
every status change stayed on the writer. `removeSchema()` and store clear or 
truncate published only with `task.sync_deletion=false`. Now every committed 
add, update, remove and store clear or truncate publishes a clear event after 
its write. Every event clears at once; events are not delayed or merged.
      A schema write is two or three PD puts. If it fails after the first one, 
PD holds part of the change. That case publishes too, and the writer drops its 
own copy of the element; the caller still gets the error.
   2. **Cache invalidation on PD watch start and recovery.** Events are 
delivered at most once, and `KvClient` subscribes again one second after a 
disconnect without telling the listener. `KvClient.listen()` has a new overload 
with a callback that runs each time a watch goes live (the first `Starting` or 
`Started` frame confirming registration on a new stream, including after a 
leader change or a stream error; an `Alive` heartbeat alone does not confirm 
registration). The Server clears all its V2 schema caches when it runs, so 
nothing missed while the watch was down survives.
   3. **Failed-publication retry.** A graph whose publish failed goes into an 
in-memory pending set and is published again after 1 s, doubling up to 30 s, 
and at once when the watch is live again. Each Server also publishes once when 
it opens a graph, which repairs a publish lost to a restart of the writer.
   4. **Generation guard.** A generation counter and a read-write lock on the 
shared cache. Every clear, removal and local write advances the generation as 
the last step of its change. A read by id, a read by name, a full listing or 
the writer's own write-through that started before a clear cannot cache its 
value after the clear. A name miss now triggers one reload per graph at a time 
instead of one per waiting thread.
   
   Plumbing for the callback: `MetaDriver.listen(key, consumer, onStarted)` as 
a default method that falls back to the plain listen, overridden in 
`PdMetaDriver`, and passed through `MetaManager` and `GraphMetaManager`. The 
etcd meta driver uses the fallback and gets no watch-start clear.
   
   ### Changed behavior of schema calls
   
   Once the schema write has committed, a failed event publication no longer 
fails the schema call. The call returns success and the publication is retried 
in the background (item 3).
   
   Before this PR the add path threw after the commit when the publish failed, 
so the caller saw an error for a change that was already stored. Inside an 
index rebuild job the same throw would fail the job after its status write had 
committed. A publish failure is logged at WARN with the graph name.
   
   What a caller can now observe: the call succeeds while other Servers still 
hold the old copy until the retry goes through or their watch restarts.
   
   ### How this relates to the review of #3237
   
   That review asked for a notification design in place of polling and listed 
points to settle first. This is where each one stands here. Nothing from #3237 
itself is revived.
   
   | Point from that review | In this PR |
   |---|---|
   | Notify, do not poll | Done. Nothing runs while the schema is idle. A retry 
timer exists only while a publish is pending. |
   | Restore update and status notifications | Done, item 1. This reverses the 
choice in #3011 to keep status changes local. The cost is measured under "Cost 
of an event". |
   | Retry without blocking DDL | Done, item 3. |
   | A reliable link between a schema commit and its notification | Partly. A 
failed publish is retried, a restarted writer publishes again, and a write that 
fails partway publishes too. The write and the event are still separate PD 
puts: see the limits. |
   | A Server that was away reloads before it serves | Partly. It drops all its 
cached schema when its watch is live again. It is not kept from serving while 
it is cut off: see the limits. |
   | Batch redundant invalidations within one operation | Not done. One index 
creation publishes four events. Merging them delays the clear of the base 
label, which widens the index window described in the limits. |
   
   ### Cost of an event
   
   One event makes every other Server drop the schema cache of that graph. Its 
next read by name then loads the whole schema again (four prefix scans in PD).
   
   Measured on 1 PD + 1 Store + 3 Servers on one host (12 CPUs), fresh cluster 
with 300 property keys, 2026-11-13: 200 updates in a row through S0 while S1 
and S2 each serve schema reads by name on two threads. Two runs per build.
   
   | | master `8828d8238` | this PR |
   |---|---|---|
   | One update on S0, mean | 7.8 and 8.0 ms | 11.8 and 11.8 ms |
   | Reads per second on S1 and S2 while idle | 3,900 and 4,140 | 3,850 and 
4,310 |
   | Reads per second during the burst | 3,610 and 3,760 | 320 and 340 |
   | Median read during the burst | 1.0 and 1.1 ms | 10.6 and 10.8 ms |
   | Reads per second right after the burst | 4,780 and 4,890 | 4,950 and 4,970 
|
   | Failed reads | 0 | 0 |
   | S1 and S2 show the last update 3 s after the burst | no | yes |
   
   So for as long as schema changes arrive back to back, schema reads on the 
other Servers were about 11 times slower in this setup, and they recovered when 
the burst ended. The cost grows with the schema: the same burst on a graph with 
about 950 property keys gave 180 to 190 reads per second. Ten index creations 
in a row cost about the same on both builds (140 to 149 ms per creation and 
2,450 to 2,580 reads per second on the other Servers). I did not measure the 
effect on graph reads and writes, only on schema reads. Lowering this cost 
means sending which element changed, or merging events, and is proposed as 
follow-up.
   
   ### What this PR does not implement
   
   These are known limits. They are listed in `docs/server-runtime.md` as 
limits, and none of them is implied to be solved by this PR. Follow-up issues 
so far: the existing #3376, and #3478 for a watch that stops for good.
   
   | Not implemented | What can still go wrong | Why it is excluded |
   |---|---|---|
   | ACK or barrier before an index rebuild | A write on another Server that is 
prepared from the old base label and committed after the rebuild scan has 
passed it is missing from the new index. `GraphTransaction.prepareCommit()` 
takes the index list from the cached label, and `IndexLabelRebuildJob` scans 
the label once and starts as soon as the task is scheduled. | The rebuild would 
have to wait until every Server has confirmed the new base label. That needs 
Server membership, confirmations and a rule for a Server that never answers. It 
is a protocol of its own and belongs in a separate design. |
   | Atomic schema commit plus event publication | A writer that stops for good 
between its schema write and its publish leaves the other Servers on the old 
copy until the next event for that graph or their next watch restart. If the 
writer restarts, its publish on graph open repairs this. | The write and the 
event are two PD puts. Making them one needs a PD-side change (one raft entry 
for both, or an outbox), which this PR avoids on purpose. |
   | Concurrent same-element schema changes | An update still writes the whole 
element from the writer's copy. Two Servers changing the same element at the 
same moment can lose one change. | It needs a conditional write in PD. That is 
#3376, which is +3,711 lines on its own. |
   | Offline-serving gate | A Server that cannot reach PD keeps serving its 
cached schema until it reconnects. | Refusing requests in that state trades 
availability for freshness. That is a product decision, not part of a cache 
fix. |
   | Restart of a watch that stopped for good | A PD watch that ends with a 
gRPC status the PD client does not retry (`UNAUTHENTICATED`, 
`PERMISSION_DENIED`, `UNIMPLEMENTED` and six more) is removed and not started 
again. The watch-start clear then never runs, and that Server keeps its cached 
schema until it is restarted; its log has one ERROR line for the stopped watch. 
PD does not answer a watch with these codes itself. They can come from a proxy 
between Server and PD: gRPC maps HTTP 401, 403 and 404 to three of them. | This 
is how the PD client on master behaves for every Server meta listener, not only 
this one. Starting the watch again needs a stop callback through the meta 
driver and a limit on how often a refused watch is retried. Tracked in #3478. |
   | Watch keepalive | A watch stream that dies without an error is noticed 
only when another PD call from that Server fails. | It needs keepalive settings 
on the PD client channels, which affects every user of the PD client and needs 
its own tests. |
   
   Two more points:
   
   - A reload right after a PD leader change also needs #3383, so the read 
cannot be served by a leader that has not applied its log yet.
   - The guarantees start when the last Server runs this version. A Server on 
the old version still publishes nothing on update. Index entries skipped before 
the upgrade stay missing until that index is rebuilt.
   
   Found while reviewing this change and left out because they are independent 
of it:
   
   - `IntObjectMap.set()` grows its array by one doubling only, so the first 
write of a schema id above twice the current size throws. It is on master and 
is hit by the array cache of both schema transactions. It can no longer skip a 
publish, because the publish runs in a `finally`.
   - Store nodes listen on the same clear key (`SchemaDriver`) without a 
watch-start callback. I did not examine what their cache is used for.
   
   ### Review scope
   
   This PR is meant to stay at its current size and shape. For a review finding 
I will do the following:
   
   - A bug in this patch, or something that breaks one of the four items under 
"What this PR fixes": fixed here, with the smallest change and a regression 
test.
   - Anything that needs one of the excluded items above (ACK or barrier, 
atomic commit, conditional write, offline gate, keepalive), a PD server change, 
or a refactor of code this PR does not touch: answered with the reason and 
proposed as a follow-up issue, not added here.
   
   The second commit is the result of such a pass over the first one. It fixes 
four things, each with a test: no event after a schema write that failed 
partway, a retry that could schedule itself twice, a generation that advanced 
before the caches had changed (a concurrent read could copy the old value into 
the array cache), and a listing flag that never recovered after a refused 
write-through.
   
   The third commit follows the review comments on the mirror, 
hugegraph/hugegraph#374. It changes no production code. The index test now 
fails when one of its background readers stops on an error (their results were 
never read, so the test could pass with fewer reads than it claims), and the 
stopped-watch limit above is added to `docs/server-runtime.md` with #3478 as 
its follow-up. The remaining comments were checked against the code and not 
taken: the `Response` in the reader loop is released by `readEntity`, the 
`Scanner` reads a stream that its `try` closes, a stopped watch is already 
logged at ERROR by the PD client, a null `onStarted` is already rejected by 
name in the PD client, and the shared test helper was retained. HTTP timeouts 
were initially deferred; the maintainer follow-up now sets finite timeouts on 
this test class's targets and removes the TODO.
   
   A second round of comments on the third commit initially led to no code 
change. Two points were new: isolation from an already-running retry, and the 
retry thread remaining after the pending set is empty. The maintainer follow-up 
fixes the isolation issue by invalidating the retry epoch and draining the 
executor outside the monitor before replacing the test publisher, with a 
deterministic regression test. The executor remains one idle daemon thread, 
created only after a first failed publish, consistent with other Server-core 
schedulers.
   
   ### Size before maintainer follow-up
   
   16 files, +2,692 / -121 (`git diff --shortstat 8828d8238 7df011acb`).
   
   | Part | Added | Removed |
   |---|---|---|
   | Production code (`CachedSchemaTransactionV2`, `KvClient`, meta plumbing) | 
628 | 111 |
   | Tests | 1,946 | 3 |
   | Cluster-test harness (restart one PD node in place) | 49 | 2 |
   | Docs | 69 | 5 |
   
   ## Verifying these changes
   
   - [ ] Trivial rework / code cleanup without any test coverage. (No Need)
   - [ ] Already covered by existing tests, such as *(please modify tests 
here)*.
   - [x] Need tests and can be verified as follows:
       - `CachedSchemaTransactionV2Test` (new, 36 tests): publish on every 
write path and after the write, failed and partial writes, failed publish and 
retry, watch start, and the generation guard on every cache insertion and 
write-through path
       - `KvClientTest` (10 new tests): the watch-start callback on first 
start, leader change, stream error and a replaced watch, including an `Alive` 
heartbeat arriving before registration confirmation
       - `MultiClusterSchemaCacheTest` (new, 5 tests): 3 PD, 3 Store and 3 
Servers
   
   ### Reproductions before and after
   
   Same scripts for both columns, through the REST API, 1 PD + 1 Store + 3 
Servers (S0, S1, S2) as local processes on one host (12 CPUs, Java 17), 
2026-11-13. "Before" is master `8828d8238`, "after" is the pre-maintainer PR 
revision used for these 2026-11-13 measurements.
   
   | Case | Before | After |
   |---|---|---|
   | Update `user_data` through S0 | Still missing on S1 and S2 after 30 s | 
Visible on S1 and S2 at the first check, after 1 s |
   | S0 appends `p1`, then S1 appends `p2` to the same label | PD ends with 
`[a, p2]`; a vertex write that sets `p1` returns 400 on S2 | All three Servers 
show `[a, p1, p2]`; the vertex is read back with `p1` through each Server |
   | Index status, 30 index creations through S0 | S1 and S2 keep `CREATING` in 
30 of 30 | 0 of 30 |
   | Base label, same 30 runs | Index missing from the label on S1 in 14 of 30 
and on S2 in 15 of 30; 29 of 29 vertices written through such a Server are not 
indexed | 0 of 30; 0 vertices not indexed |
   | Add through S0 right after a PD restart | Missing on S1 and S2 after 60 s, 
3 of 3 runs | Visible within 2 s, 3 of 3 runs |
   | Remove with `task.sync_deletion=true` | Still listed on S1 and S2 after 30 
s | Gone on all Servers within 1 s |
   
   In the "after" run each Server logged the watch-start clear four times: once 
at start and once per PD restart. No Server logged a failed publish.
   
   All of this ran on one host with sub-millisecond latency. It shows that the 
cases happen and stop happening, not how often they occur in production.
   
   ### Author test suites before maintainer follow-up
   
   Java 17, 2026-11-13. Format and multi cluster ran on the tree of the head 
commit. The other suites ran on the second commit; the third changes one 
cluster test and one doc file and adds no file. Server unit and HStore core ran 
before one doc paragraph and one comment in the cluster-test harness were added 
to the second commit.
   
   | Suite | Command | Result |
   |---|---|---|
   | Format | `mvn editorconfig:check -pl 
hugegraph-pd/hg-pd-test,hugegraph-server/hugegraph-test,hugegraph-cluster-test/hugegraph-clustertest-test
 -am -ntp` | BUILD SUCCESS |
   | Licence headers | `mvn apache-rat:check -ntp` | BUILD SUCCESS |
   | Server unit | `mvn test -pl hugegraph-server/hugegraph-test -am -P 
unit-test -ntp` | 1,477 run, 0 failures, 8 skipped |
   | HStore core (PD and Store running) | `mvn test -pl 
hugegraph-server/hugegraph-test -am -P core-test,hstore -ntp` | 1,033 run, 0 
failures, 77 skipped |
   | Multi cluster | `mvn test -pl 
hugegraph-cluster-test/hugegraph-clustertest-test -am -P multi-cluster-test 
-ntp` | 11 run, 0 failures (5 are `MultiClusterSchemaCacheTest`); the runtime 
log check passes. Run again on the third commit with the same result |
   | PD client (PD running) | `mvn test -pl hugegraph-pd/hg-pd-test -am -P 
pd-client-test -ntp` | 122 run, 0 failures (`KvClientTest`: 48), run on the 
first commit; historical evidence, superseded for the modified PD client by the 
follow-up validation below |
   
   The two cache test classes were also run 12 times in a row in fresh JVMs: 12 
of 12 passed.
   
   One existing test changed: 
`CachedSchemaTransactionTest.testV2WarmRequestsRetainSchemaWithoutMetadataReadsAndReceiveInvalidations`
 expected one PD put and now expects one at graph open and one more for the 
store clear.
   
   ### Do the tests detect the bugs
   
   - `MultiClusterSchemaCacheTest` on unpatched master `8828d8238`: 4 of 5 fail 
(update, append, index creation, PD restart).
   - Each fix was taken out in turn and `CachedSchemaTransactionV2Test` was run 
against it:
   
   | Change taken out | Tests that fail |
   |---|---|
   | Publish after the write (publish moved before it) | 
`testEveryPublishComesAfterItsStoreWrite`, `testAddPublishes` |
   | Handling of a write that failed partway | 
`testWriteThatFailsHalfwayPublishesAndDropsTheElement`, 
`testRemoveThatFailsHalfwayPublishesAndDropsTheElement`, 
`testFailedWriteKeepsItsEventWhenThePublishFailsToo` |
   | Only the newest retry reschedules | 
`testWatchStartDuringARetryLeavesOneRetry` |
   | Generation advanced last in a local write | 
`testReadDuringALocalWriteDoesNotPutTheOldValueBack` (3 of 3 runs) |
   | Eviction after a refused write-through | 
`testRefusedWriteThroughEvictsWhatWasCachedMeanwhile`, 
`testReadsNeverGoBackBehindAWriteUnderConcurrentClears` |
   | Read lock around a cached listing | 
`testAClearWaitsForAListingReadFromTheCache`, 
`testListingIsNeverPartialWhileClearsRun`, 
`testReadsNeverGoBackBehindAWriteUnderConcurrentClears` |
   | Whole-list flag set after a full load | 
`testRefusedWriteThroughEvictsWhatWasCachedMeanwhile` |
   
   ### Known gaps in the evidence
   
   - `testRemoveReachesEveryServerWithSyncDeletion` passes on unpatched master, 
because a new graph publishes events while its system schema is set up. The 
unit test and the manual check in the table above cover that gate.
   - `testChangeDuringPDRestartReachesEveryServer` does not prove that a watch 
was down when the change committed. It failed on unpatched master in the one 
control run. The evidence for the watch-start clear is the unit tests plus the 
Server log lines of the manual run.
   - `testReadsNeverGoBackBehindAWriteUnderConcurrentClears` and 
`testListingIsNeverPartialWhileClearsRun` depend on timing. The first one 
failed in about half of the runs before the generation was advanced last, and 
in none of 12 after.
   - The generation order is tested with a deterministic test for a local 
write. For a clear it is covered only by the timing test above.
   - The publish in `finally` after a failure inside the cache update has no 
test.
   - The in-flight write window during index creation and the writer that never 
returns are derived from the code and were not reproduced.
   - Not tested: a real network partition, a silent stream death, Servers on 
mixed versions.
   - Not rerun locally for the maintainer follow-up: api-test, the TinkerPop 
suites, the PD core, common and REST suites, the Store suites and the 
simple-cluster suite. Historical CI and current-head validation are recorded 
separately below.
   
   ### Maintainer follow-up and validation: `d2d54fe`
   
   Reviewed from three independent angles (cache correctness, watch/retry state 
transitions, and existing comments/test quality), followed by adversarial 
cross-checks. The follow-up changes 5 files, +156 / -15 relative to 
`7df011acb`; only +15 / -7 are in production code. It adds no PD-server/proto 
change, polling, ACK/barrier, outbox, or extra retry abstraction.
   
   - Separate receipt of a transport frame from confirmed watch registration; 
run the started callback only on the first `Starting` / `Started` of the 
current attempt.
   - Assert HTTP 2xx in background readers, and set this cluster test's targets 
to a 5 s connect timeout and 15 s read timeout.
   - Invalidate the retry epoch and wait for in-flight test publication to 
finish before replacing the test publisher.
   - Document that DDL for the same schema element must be serialized through 
one pinned Server; verify visibility before changing writers. Remove the 
unconditional no-request-failure claim.
   
   Validation on the exact tree pushed as `d2d54fe`, Java 17, 2026-11-15 
(Asia/Singapore):
   
   | Validation | Result |
   |---|---|
   | Full 38-module clean compile, packaging, editorconfig format/check, and 
diff whitespace check | Passed |
   | `CachedSchemaTransactionV2Test` + `CachedSchemaTransactionTest` | 36 + 21 
= **57 passed** |
   | Full `KvClientTest` against an actual PD service | **51 passed** |
   | `MultiClusterSuiteTest`, actual 3 PD + 3 Store + 3 Server processes | **11 
passed**, including all 5 schema-cache tests; runtime log check passed |
   | Total distinct positive tests in this follow-up | **119 passed; 0 
failures, 0 errors, 0 skipped** |
   | Restore old watch-start logic while retaining new regressions | All **3** 
new tests fail at their intended assertions |
   | Restore old reset helper while retaining the in-flight regression | The 
**1** new test fails at its intended assertion |
   
   The fixed code was restored and its tests passed after both mutation checks. 
Cache tests used Mockito's supported explicit startup Java agent because 
self-attachment is unavailable in this environment. Multi-node child JVMs used 
a 512 MiB heap cap, 2 active processors, and 128 MiB direct-memory/RocksDB 
limits; the test configuration uses Store replica count 1. These are functional 
checks, not a performance benchmark or evidence for three-replica behavior. The 
author's earlier throughput measurements were not rerun.
   
   The successful [ASF workflow run 
43432139802](https://github.com/apache/hugegraph/actions/runs/43432139802) 
applies to `7df011acb`, the head of #3477. The CI of this PR is the first ASF 
run on `d2d54fe`.
   
   The permanent stopped-watch recovery gap remains tracked by #3478. Strong 
concurrent DDL protection, an index-rebuild barrier, atomic schema/event 
publication, and offline-serving policy remain separate design work.
   
   ## Does this PR potentially affect the following parts?
   
   - [ ]  Dependencies ([add/update 
license](https://hugegraph.apache.org/docs/contribution-guidelines/contribute/#321-check-licenses)
 info & 
[regenerate_known_dependencies.sh](../install-dist/scripts/dependency/regenerate_known_dependencies.sh))
   - [ ]  Modify configurations
   - [ ]  The public API
   - [x]  Other affects: a schema call no longer fails when only the event 
publication failed after the commit (see "Changed behavior of schema calls"). 
During a burst of schema changes, schema reads on the other Servers slow down 
(see "Cost of an event"). New Java methods: a `KvClient.listen()` overload with 
a started callback and a default method `MetaDriver.listen(key, consumer, 
onStarted)`. Each Server publishes one clear event per graph when it opens the 
graph.
   - [ ]  Nope
   
   ## Documentation Status
   
   - [ ] `Doc - TODO`: required documentation is pending; complete it before 
merging.
   - [x] `Doc - Done`: documentation is included here or linked below.
   - [ ] `Doc - No Need`: no user-visible documentation is affected.
   
   Documentation files in this PR or paired hugegraph-doc PR:
   
   - `docs/server-runtime.md`: new section "Schema caches across HStore 
Servers" with the guarantees and the limits
   - `helm/hugegraph/docs/operations.md`: the paragraph on index status across 
Pods
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to