arshilkxwork opened a new pull request, #3357:
URL: https://github.com/apache/hugegraph/pull/3357
## Purpose of the PR
- part of #3136 (the PD-side gauges; the Store-side families stay open, and
the issue author offered to take those)
PD keeps every store's state and every shard group's leader in its metadata,
but its Prometheus endpoint only exports how many stores exist. While a rebuilt
Store recovers, an operator has to read logs or poll `/v1/stores` and
`/v1/shardGroups` to see whether it is back `Up` and whether leaders have moved
back to it. Because PD serves these gauges, they stay readable while the
recovering Store cannot serve its own endpoint.
## Main Changes
- New `StoreStateMetrics` in `hg-pd-service`, registered from `PDMetrics`:
- `hg_store_state{store,state}` is `1` for the store's current
`StoreState` and `0` for the other five values.
- `hg_store_leader_count{store}` counts the shard groups whose `Leader`
shard is on the store. A store that leads nothing reports `0`.
- Stores join and leave at runtime, so the meters follow the store list. A
refresh reads `getStores()` and `getShardGroups()` once, registers gauges for
new store ids and removes the gauges of ids PD no longer has. The gauges
trigger the refresh themselves at most once a second, so one scrape reads the
metadata once. The existing `hg.stores` gauge also triggers it, which is how
the first gauges of a new store get registered. The same pattern is already
used by `hg.graphs`, which registers per-graph meters from its callback.
- A failed read (`PDException`, or a scrape that arrives before `PDService`
has created its `StoreNodeService`) logs a warning and keeps the previous
values.
- `hugegraph-pd/docs/api-reference.md`: a "Store state gauges" section with
both families, the one-scrape delay for a new store, and a note to select one
PD instance before aggregating, since every member reads the same replicated
metadata.
Cardinality is seven series per registered store (six states plus the leader
count).
## Verifying these changes
- [ ] Trivial rework / code cleanup without any test coverage. (No Need)
- [ ] Already covered by existing tests, such as *(please modify tests
here)*.
- [x] Need tests and can be verified as follows:
- `StoreStateMetricsTest` in `hg-pd-service/src/test/java`, next to
`BalanceLeadersAPITest`: state and leader counts for three stores with one
`Offline`; stores joining and leaving add and remove their gauges; a failing
read keeps the last values; and a `PrometheusMeterRegistry` scrape, with
refresh driven from another gauge the way `hg.stores` drives it, prints
`hg_store_state` and `hg_store_leader_count` with the expected values.
- `mvn test -pl hugegraph-pd/hg-pd-service`: 21 tests, 0 failures.
- The test sits in `hg-pd-service` because that module is packaged as a
Spring Boot jar, so `hg-pd-test` cannot load its classes. CI runs it through
`-am` on the PD jobs.
- Not run against a live PD with registered Stores. The PD REST CI job
starts PD without a Store, so `RestApiTest` would see no per-store series and I
did not add an assertion there.
## Does this PR potentially affect the following parts?
- [ ] Dependencies ([add/update
license](https://hugegraph.apache.org/docs/contribution-guidelines/contribute/#321-check-licenses)
info &
[regenerate_known_dependencies.sh](../install-dist/scripts/dependency/regenerate_known_dependencies.sh))
- [ ] Modify configurations
- [ ] The public API
- [x] Other affects (typed here): new Prometheus series on PD's
`/actuator/prometheus`
- [ ] Nope
## Documentation Status
- [x] `Doc - TODO`: required documentation is pending; complete it before
merging.
- [ ] `Doc - Done`: documentation is included here or linked below.
- [ ] `Doc - No Need`: no user-visible documentation is affected.
Documentation files in this PR or paired hugegraph-doc PR:
- `hugegraph-pd/docs/api-reference.md` (in this PR)
- The PD page on the website lists the `hg_*` meters PD exports; a paired
apache/hugegraph-doc PR will add these two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
