SebastianGruza opened a new pull request, #3221:
URL: https://github.com/apache/hugegraph/pull/3221
## Purpose of the PR
Closes #3212. A Kubernetes readiness probe on `/versions` keeps a Server in
the Service while every graph request fails, because `/versions` answers 200 as
long as the REST layer is up, even with no Store in the cluster (measured under
#3132: `ready=true` with 0 Stores for 150+ s while every `GET
/graph/vertices/<id>` ended in a 500 after the 30 s request bound). Design
discussed in the issue.
## Main Changes
`GET /readiness` answers 200 while this Server can serve graph traffic and
503 otherwise, unauthenticated like `/versions` (whitelisted in
`AuthenticationFilter` and `PathFilter`, so an `httpGet` probe needs neither a
credential nor a graphspace prefix), with a JSON body that carries no addresses:
```
{"ready":true,"storage":"hstore","reason":"ok","active_stores":3,"answered_store":6386607161404282741,
"pd_reachable":true,"pd_checked_age_ms":312,"stores_age_ms":312,"store_millis":3,"cached":false}
```
Ready means, from this Server's own view: at least one Store from the last
Store list PD answered with answers a direct, local, read-only gRPC call
(`HgStoreState.getScanState`, a read of the node's own scan-pool stats that
never touches raft). The Store list is refreshed from PD in the background on
every probe and never waited for once a list is known, so PD being down, slow
or restarting only flips the reported `pd_reachable`, never the readiness, as
long as a Store answers. The pings run in parallel and the first answer wins,
so a Store whose pod just left never eats the budget of the healthy ones.
- `hugegraph-hstore`: `HstoreStorageProbe` (pure logic over a Store lister
and a Store pinger, one shared time budget through an executor so a hung PD or
Store yields "did not answer within N ms" instead of a hung probe;
`KnownStores` holds the last PD answer) and a `storage_readiness` meta handler
in `HstoreStore`.
- `hugegraph-api`: `ReadinessAPI` and `StorageReadiness` (first hstore graph
is probed as the internal admin through `HugeGraph.metadata(null,
"storage_readiness", timeout)`, result cached for `readiness.cache_ttl`;
Servers with no hstore graph answer 200 with `storage=embedded`); the api
module gains no dependency.
- `ServerOptions`: `readiness.timeout` (default 1000 ms) and
`readiness.cache_ttl` (default 2000 ms).
## Verifying these changes
- [x] Add new unit test(s): `HstoreStorageProbeTest` (15: first answer wins,
a hung Store does not hide an answering one, known Stores keep the Server ready
while PD is down, a hung PD does not delay a probe with known Stores, the PD
answer updates the known list, budget bounding, no addresses in the body) and
`StorageReadinessTest` (6, including the two filter whitelists).
- [x] E2E on a three-node k3s with the chart from #3218 /
hugegraph/hugegraph#221, 1 Hz sampling of `/readiness`, `/versions`, the pod
Ready condition and the Service endpoints on every Server pod, load through the
Service: **7 PASS** (baseline; Stores→0 pulls every Server out of the Service
in ~30 s and back in ~20 s; one Store deleted, no flap; **rolling restart of
the Stores under load, zero readiness transitions**; PD→0 stays 200 with
`pd_reachable=false`; rolling restart of PD, zero transitions; SIGSTOP the
busiest Store, zero transitions). Scripts, per-scenario JSON and the two
superseded probe designs (an `existsTable` ping flapped on a roll; probing PD
on every request tracked PD instead of storage):
https://github.com/SebastianGruza/hugegraph-validation/tree/master/results/issue-3212
## Does this PR potentially affect the following parts?
- [x] Introduce new configurations: `readiness.timeout`,
`readiness.cache_ttl` in `ServerOptions`
- New unauthenticated endpoint `GET /readiness`; no data-format change.
## Chart side
A companion change adds `server.readinessPath` (default `/versions`) to the
#3132 chart, mirroring `pd.readinessPath`; set it to `/readiness` on an image
that serves it. Diff and measurements are in the validation repo above and will
go to hugegraph/hugegraph#221.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
