bitflicker64 commented on PR #3185: URL: https://github.com/apache/hugegraph/pull/3185#issuecomment-5550447092
Tested end to end on Kubernetes on 2026-09-05, with this branch merged into the hugegraph/hugegraph testing tree. **Build under test.** Tag [`helm-dev-20260905`](https://github.com/hugegraph/hugegraph/releases/tag/helm-dev-20260905), commit `6ec19838`: apache master `36811483` plus this PR at `70744406`, #3187 and #3189. PD, Store and Server images built from that tree with `docker buildx bake -f docker/bake.hcl pd store server-hstore` (linux/amd64), labelled `org.opencontainers.image.revision=6ec1983889f58a52995b27a4e1ea89bd00932396`, and the label read back from every running pod. kind v0.33.0, Kubernetes 1.37.0, 1 control-plane + 3 workers, the #3132 Helm chart with `pd.readinessPath=/v1/ready` and `store.waitPath=/v1/ready`, 3 PD + 3 Store + 3 Server + Hubble. **What held.** Deleting two of three PDs with `kubectl delete pod`, the survivor sampled every 2 s through a port-forward, two runs: - `/v1/ready` on the survivor: 503 with `hg_raft_has_leader 0` 2.6 s after the delete, back to 200 with `hg_raft_has_leader 1` once the replacements rejoined, 12 to 20 s later. - `/v1/health` on the same pod: 200 on all 313 samples of run one and all 90 of run two, so liveness and readiness now disagree exactly when they should. - A Store deleted inside the window sat in its `wait-for-pd` init container (`Waiting for 2 PD peers to answer /v1/ready`) from T0+7 s and passed at T0+23 s, the same second the first 200 came back. Schema writes through the Server hung inside the window and returned 202 after it. - The gauges behaved as documented: `hg_raft_alive_peers` read 3.0 while the sampled node was leader and NaN once leadership moved to a replacement. **One thing worth a look before merge: the handler stalls during the election.** With `curl -m 10 -w '%{time_total}'`, the first `/v1/ready` request after the delete took 9.79 s to return its 503; every 200 before and after answered in under 5 ms. In the first run, with a 2 s client timeout, every sample inside the leaderless window except the first came back as a timeout rather than a 503. A kubelet probe with `timeoutSeconds: 5` fails either way, so the readiness contract is met, but the endpoint promises a prompt 503, not a stall. Reading the path: `StoreAPI.checkReady` calls `RaftEngine.getRaftStatus`, which reads `node.getNodeState()` and `node.getLeaderId()`, both under `NodeImpl`'s lock; while the node runs its election it holds the write lock and opens connections to two peers that no longer exist. I have not profiled it, so treat the lock as the lead and the 9.79 s as the measurement. One way out is to keep a volatile copy of the last known state and leader, updated from the state machine callbacks (`onLeaderStart`, `onLeaderStop`, `onStartFollowing`, `onStopFollowing`), and serve the probe from that, so it never waits on the node lock. The sample logs and scripts are kept with the campaign notes; I can attach them here if useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
