SebastianGruza commented on issue #3212:
URL: https://github.com/apache/hugegraph/issues/3212#issuecomment-5734240416
I'm taking this one, as agreed with the chart author. Design below so it can
be rejected before the PR; the code exists locally, the E2E on k3s is running,
and the PR will come with the measurements.
### Endpoint
`GET /readiness`, unauthenticated (added to `FIXED_WHITE_API_SET` next to
`versions`), 200 or 503, JSON without addresses:
```
200
{"ready":true,"storage":"hstore","reason":"ok","active_stores":3,"answered_store":1844095761336398711,
"pd_millis":4,"store_millis":6,"cached":false}
503 {"ready":false,"storage":"hstore","reason":"no active store registered
in pd","active_stores":0,
"answered_store":null,"pd_millis":3,"store_millis":0,"cached":false}
```
A separate path rather than a flag on `/versions`, because `/versions` is
the chart's startup and liveness probe today and should stay "the REST layer is
up"; readiness has a different meaning and a different blast radius.
`/apis/readiness` from the proposal is out, since `/apis` belongs to ProfileAPI.
### Definition of "ready"
1. This server's PD client answers: `PDClient.getActiveStores()` within the
time budget.
2. At least one of the stores PD reports as active answers a direct, cheap
call from this server: a node session through `HgStoreNodeManager` and
`existsTable("g+v")`, i.e. one `Table/EXISTS` RPC that touches no data.
Answering counts, not the answer. Stores are tried in turn; the first one that
answers ends the probe.
Why "at least one" and not "all": readiness answers whether this server can
serve traffic, not whether the cluster is healthy. During a rolling restart of
the stores one is always on its way; requiring all of them would pull every
server out of the Service on every roll, which is exactly the flapping the
chart author asked about. The zero-store case from #3132 is caught either way.
PD's view can be stale (measured: PD showed a store as Up 200 s after `kill
-9`), hence step 2 is mandatory.
### Cost and bounding
- One shared budget for the whole probe, `readiness.timeout` (ms, default
1000): PD and each store get the remaining budget, calls go through an executor
with `Future.get(remaining)`, so a hung PD or store yields a 503 with "did not
answer within N ms", never a hung probe.
- The result is cached for `readiness.cache_ttl` (ms, default 2000):
kubelet, a load balancer and a human with curl share one PD call and one store
RPC per window. 0 disables the cache.
- The storage is shared by every hstore graph in the process, so one graph
is probed; a server with no hstore graph (rocksdb, tests) answers 200 with
`"storage":"embedded"`.
### Where it lives
- `hugegraph-hstore`: `HstoreStorageProbe` (pure logic over two functional
interfaces, the active-store list and the ping, plus the wiring through
`HstoreSessionsImpl.getDefaultPdClient()` and `HgStoreNodeManager`);
`HstoreStore` registers a `storage_readiness` meta handler.
- `hugegraph-api`: `ReadinessAPI` (`@Path("readiness")`, `@PermitAll`) and
`StorageReadiness` (graph selection, cache, mapping to 200/503), called through
the existing `HugeGraph.metadata(null, "storage_readiness", timeout)` hook. The
api module does not depend on hstore or on the store client, and stays that way.
- `ServerOptions`: `readiness.timeout`, `readiness.cache_ttl`.
### Tests
- Unit: `HstoreStorageProbeTest` (9: PD fails, PD hangs within the budget,
zero stores, every store fails, first answering store after two failures, hung
stores within the budget, no addresses in the result), `StorageReadinessTest`
(5: cache, TTL 0, probe exception as 503 with a reason, budget passed through,
copy isolation).
- E2E on a three-node k3s with the chart from #3218 /
hugegraph/hugegraph#221, with the same harness as the 2026-09-16 battery
(oracle before the fault, load through the Service, 1 s sampling of
`/readiness`, `/versions`, the Ready condition and the Service endpoints on
every server pod): baseline, stores → 0 and back, one store deleted, rolling
restart of the stores under load, PD → 0 and back, rolling restart of PD,
SIGSTOP on the store leading the most partitions. Criteria: with stores → 0
every server leaves the Service; on every roll not a single Ready → NotReady
transition. Results go into the PR and to hugegraph/hugegraph#221.
### Chart side
`server.readinessPath` defaulting to `/versions`, switched to `/readiness`
on images that serve it, exactly like `pd.readinessPath`; startup and liveness
stay on `/versions` so a server that merely lost its storage is not restarted.
The diff for #221 is ready and will go with the measurements.
What I'm not doing: probing every graph and every store, checking partition
leaders, an authenticated variant of the endpoint. If any decision above is
wrong, now is the cheapest moment to say so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
