SebastianGruza commented on PR #3157:
URL: https://github.com/apache/hugegraph/pull/3157#issuecomment-5530351739
Live before/after for this branch (head `f99b6bd`) on a real cluster: PD + 3
store nodes, two HugeGraph servers built from the same tree sharing the PD in
PD-meta mode (`usePD=true`), server A on :8080 and server B on :8082. Server
distributions built from master `98477f0` (before) and from this head (after);
PD and stores unchanged.
**Observable that lazy loading cannot compensate:** in PD-meta mode
`GraphManager.graph()` loads an unknown graph from PD meta on first access, so
"B sees a graph created on A" proves nothing about the watch. Removal is
different — B keeps a graph it has already constructed until the `GRAPH/REMOVE`
event arrives. So each probe is: A creates `g` → B constructs it on first
access (HTTP 200) → A drops `g` → poll B's `GET …/graphs/g/schema/propertykeys`
until it stops answering 200. Between probes PD is stopped for 30 s (SIGTERM
via `stop-hugegraph-pd.sh`, then `kill -9` of the port owner) and restarted;
the stores stayed up throughout.
| Step | master `98477f0` | this branch |
|---|---|---|
| baseline: create → drop | B dropped `g1` after **1 s** (`Accept graph add
signal` seen once) | B dropped `g1` after **1 s** |
| PD down 30 s, back, settle 15 s → create → drop | B **still serves the
dropped `g2` after 60 s** (HTTP 200); no add signal in B's log → watch dead | B
dropped `g2` after **1 s**; add signal seen |
| second PD outage → create → drop | B **still serves `g3` after 60 s**;
watch dead | B dropped `g3` after **1 s**; add signal seen |
B's log on this branch during each outage: `Failed to reconnect watch for
key HUGEGRAPH/hg/EVENT/GRAPH/{ADD,REMOVE,UPDATE,CLEAR,SCHEMA/CLEAR}` about once
per second per subscription, then `AbstractClient connect to host = … success`
and five fresh `set watch client id` lines within a second of PD listening
again. On master B logs nothing after the first `set watch client id` — the
single re-`listen()` from `onError` never comes back once PD is down.
So the failure described in #3152 reproduces on master with a plain 30 s PD
restart, and this branch recovers every metadata watch after it, twice in a row.
Two small remarks, neither blocking:
- The retry path logs a WARN **with a full stack trace** for every attempt:
with the 4 graph-event subscriptions plus one schema-cache-clear subscription
per open graph, server B wrote ~210 `Failed to reconnect watch for key …`
warnings (each with a trace) per 30 s outage. A one-line WARN, or the trace
only on the first failure of a subscription, would keep this readable during a
longer PD outage.
- Unrelated to this PR but hit while setting the test up: with `usePD=true`
the server does not start on a JVM whose default locale uses a decimal comma
(`GraphSpace.info()` does `Float.parseFloat(String.format("%.2f", …))`, `pl_PL`
→ `"0,00"` → `NumberFormatException`). I'll file it separately.
Scripts and full logs (both sides, both variants):
https://github.com/SebastianGruza/hugegraph-oracle-suite —
`cluster/pd_watch_exp2.sh`, `results/pd_watch_3152_*.log`. Happy to re-run on
the next head — ping me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
