SebastianGruza commented on PR #3132: URL: https://github.com/apache/hugegraph/pull/3132#issuecomment-5701778807
An independent run of the chart on a three-node k3s (`v1.36.4+k3s1`, 3 × 6 vCPU / 16 GB, Debian 13, `local-path`), this PR's branch at `895019159439a7ea0d3a35c20e656ac53c554004` (chart 0.1.0), `:latest` images from Docker Hub: `pd@sha256:e5f72158849301e194c6ab939f8f8111e4da544000a69f890dfe8daa2b068470`, `store@sha256:0e0623a59ced0aefe0e5680219d52fba50c3568712186180fc84f114113c4929`, `server@sha256:5e437813753122c970949fea627105eeb65fec3255fb2130438f9d98372e7127`. Both presets come up clean: `values-single.yaml` in 98 s with `helm test` green, `values-cluster.yaml` 3+3+3 in 77 s, about 3.8 GB RAM per node at rest. The rest is the fault scenarios from my VMs moved onto pods: 20 000 vertices and 100 000 edges as the oracle (400 vertices sampled with their exact edge sets after every scenario, every write acknowledged with HTTP 201 during a fault read back afterwards), a writer and a reader running throughout, and evidence that the fault landed (pod uid, restart count, process state `T`, PD leader change). Full write-up, scripts and per-scenario JSON: `docs/helm-chart-faults.md` and `results/helm-chart/` in https://github.com/SebastianGruza/hugegraph-validation. | # | scenario | what happened | verdict | |---|---|---|---| | 1 | Store pod force-deleted | new uid, Ready after 23 s; writes 354 / 0 failed, max 5.2 s; reads 0 failed | PASS | | 2 | Store JVM `SIGSTOP` 120 s, victim leading 0 partitions | liveness killed the container after ~50 s; 0 failures, max 0.5 s | PASS, but silent: a follower-only Store stall is invisible to clients | | 3 | Store JVM `SIGSTOP` 120 s, victim leading 14 of 24 partitions | liveness kill after 63 s, leaders moved to the other two Stores (12/12); writes 352 / 0, max 4.3 s; reads 353 / **1**: one `GET` hit the 30 s request timeout and came back `500 Interrupted while waiting to retry` (the #3204 path) | PASS: the F15 shape that held a REST worker for up to 19 min on VMs ends on pods with one 30 s request and a restart after 63 s | | 4 | PD leader pod force-deleted | new leader after 31.9 s, pod Ready after 9.5 s; 1 write hit the 30 s timeout; a schema element created through server A visible through B and C | PASS | | 5 | PD leader JVM `SIGSTOP` 90 s, default `raft.rpc-timeout` (10 000) | **no PD leader from 12.8 s to 56.2 s** (every sample), new leader at 60.6 s, i.e. exactly when liveness killed the frozen PD; writes 680 / 0, reads 0 | PASS for the data plane; the control plane had no leader for a minute (your 67 s with a blackhole) | | 6 | same with `pd.javaOpts="-Draft.rpc-timeout=3000"` | **new leader after 8.5 s**, zero leaderless samples; writes 671 / 0, max 3.4 s | PASS: your untested suggestion works, 60 s → 8.5 s | | 7 | existing cluster: Stores scaled to 0, Servers rolled | Servers `Running`, **`ready=true`**, 0 restarts after 150 s with no Store at all; `/versions` 200 while `GET /graph/vertices/<id>` → 500 after 30 s | PASS for the no-exit rule; readiness finding below | | 8 | fresh install, Stores scaled to 0 within 1 s of `helm install` (the #3203 shape) | Servers `ready=false` for 244 s, 0 restarts, `[wait-storage] No Up store yet` every 5 s in the log; Stores back → Ready 20 s later, schema and a vertex written | PASS: the chart's gate holds the Server where the bare image exited 1 | 8 PASS, 0 FAIL, 0 INCONCLUSIVE; no acknowledged write was lost in any scenario. Five things for the chart: 1. **Readiness does not reflect the data plane.** Server readiness is `GET /versions`, which answers 200 with no Store in the cluster (scenario 7), so the Service routes to Servers that end every graph request with 500 after 30 s. A path that touches storage (the graph list from PD plus one cheap Store call) would take such a Server out of the Service. Blame: chart (probe choice). 2. **`raft.rpc-timeout=3000` should be the PD default in the chart.** With the image default a frozen leader leaves the cluster without a PD leader for ~60 s and the new one only appears once liveness kills the frozen PD; with 3000 ms the election takes 8.5 s. Data was unaffected in both runs, but everything that needs PD (schema, new partitions, Server registration) waits the full minute. 3. **Liveness is what ends a stall.** For a Store frozen with `SIGSTOP`, `/v1/health` fails and kubelet restarts the container after 50-63 s. The #3204 bound showed up as exactly one 30 s request; the rest of the load never noticed because raft moved the 14 leaders within seconds. Worth one sentence in the README: the 20 s × 3 liveness budget is the upper bound of a stalled Store's blast radius. 4. **`values.schema.json` refuses `store.replicas=0`**, so "PD and Servers first, Stores later" cannot be expressed in values; the cold-start test had to scale the StatefulSet right after install. Fine as a guard rail; the README could say how to stage a rollout. 5. **Naming**: for a release not named `hugegraph`, workloads and Services are `<release>-hugegraph-*`, while the Secrets are `<release>-admin`, `<release>-auth-token`, `<release>-pd-auth`. Cost me one failed check; cosmetic. Not covered yet: the snapshot/compaction race (#3162) on pods (needs the Store test endpoints and unflushed data, as on the VMs), a full PD disk on `values-single.yaml`, a rolling upgrade under load, and a NetworkPolicy variant of the blackhole (the chart has none yet). If you want either of points 1-2 as a change in the chart, I can prepare a PR against your branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
