bitflicker64 opened a new issue, #3136:
URL: https://github.com/apache/hugegraph/issues/3136

   ## Feature Description (功能描述)
   
   ### Problem
   
   When a Store node is rebuilt after data loss and rejoins the cluster, its 
shards recover through raft snapshot install plus log catch-up. On a graph of 
any real size this takes minutes. Under the current implementation, that 
progress is visible only in log lines: no exported metric reports how many 
partitions are still syncing, how far along a snapshot install is, or how far 
the local raft log trails the leader. This makes it hard for an operator to 
distinguish a healthy long recovery from a stuck one, or to estimate remaining 
time.
   
   Operational observation from my own fault testing (not a code claim): after 
deleting a Store pod and its volume in Kubernetes and letting the StatefulSet 
rebuild it, the only available signals were pod readiness and log lines such as 
`Raft {} begin loadSnapshot` / `Raft {} end loadSnapshot`. Nothing on the 
Prometheus endpoint moved in a way that reflected recovery progress.
   
   ### What the Store exports today (verified on master)
   
   The Store node already runs a Spring actuator Prometheus endpoint. 
`hg-store-dist/src/assembly/static/conf/application.yml` (lines 22-30) enables 
the Prometheus meter registry and exposes all web endpoints, so 
`/actuator/prometheus` is served on the REST port. Meter families are 
registered in 
`hg-store-node/src/main/java/org/apache/hugegraph/store/node/metrics/MetricsConfig.java`
 (lines 32-46), which adds the common tag `hg=store` and initializes five 
sources:
   
   - `StoreMetrics` (`store/node/metrics/StoreMetrics.java`, lines 51-79): 
`hg.up`, `hg.graphs`, and `hg.partitions{graph=...}` which is only a count of 
partitions per graph.
   - `JRaftMetrics` (`store/node/metrics/JRaftMetrics.java`, lines 73-126): 
`jraft.groups` plus a generic re-export of whatever sofa-jraft `NodeMetrics` 
registers per raft group (latency timers, replicator counters, tagged 
`group=<id>`). These are low level jraft internals; nothing in them is 
partition aware, and nothing expresses "this group is installing a snapshot" or 
"this group is N entries behind".
   - `RocksDBMetrics` (`rocks.stats.*` prefix, `RocksDBMetricsConst.java` line 
32), `ProcfsMetrics` (`process_memory.*`), `GRpcExMetrics` (`grpc.*`).
   
   I read every file in the `store/node/metrics` package on master: there is no 
metric family for partition work state, shard sync state, snapshot install 
progress, raft log lag, leader count per store, or a recovering-partition count.
   
   ### Where recovery progress already lives in memory
   
   The information needed for such metrics already exists in process; it is 
just never handed to the meter registry:
   
   - Snapshot install: 
`hg-store-core/src/main/java/org/apache/hugegraph/store/raft/HgStoreStateMachine.java`,
 `onSnapshotLoad` (lines 216-244) knows the group id and the snapshot's last 
included index, and 
`hg-store-core/src/main/java/org/apache/hugegraph/store/snapshot/HgSnapshotHandler.java`,
 `onSnapshotLoad` (lines 165-203) walks the snapshot data directory while 
loading it into RocksDB. Start, end, and the file set (with sizes on disk) are 
all known there; today the only output is log lines (lines 177-180).
   - Log catch-up: `HgStoreStateMachine.onApply` advances `committedIndex` on 
every applied entry (lines 105, 119-121), surfaced per group by 
`PartitionEngine.getCommittedIndex` 
(`hg-store-core/src/main/java/org/apache/hugegraph/store/PartitionEngine.java`, 
lines 718-720). The same value is already packaged per partition into 
`Metapb.RaftStats` for the PD heartbeat by 
`hg-store-core/src/main/java/org/apache/hugegraph/store/metric/HgMetricService.java`,
 `getRaftMetrics` (lines 114-124), so lag against the leader is computable from 
data the system already collects.
   - Partition and shard state: `HgMetricService.getStoreMetrics` (lines 
126-157) attaches each partition's `workState` and local shard role to the 
heartbeat, and `HeartbeatService.partitionHeartbeat` 
(`hg-store-core/src/main/java/org/apache/hugegraph/store/HeartbeatService.java`,
 lines 290-352) reports per-shard `SState_Normal` / `SState_Offline`. The proto 
already defines the exact states a recovery dashboard needs, including 
`SState_Snapshot` ("Install snapshots") in 
`hg-pd-grpc/src/main/proto/metapb.proto` (lines 251-259) and `PartitionState` 
(lines 111-121).
   - Leader count: `HgStoreEngine.getLeaderPartition` 
(`hg-store-core/src/main/java/org/apache/hugegraph/store/HgStoreEngine.java`, 
lines 536-544) already computes the set of groups this node leads. Raft metrics 
are enabled on every group (`PartitionEngine.java`, lines 199 and 244).
   - There is a JSON view of some of this (`/v1/partitions` in 
`hg-store-node/src/main/java/org/apache/hugegraph/store/node/controller/PartitionAPI.java`,
 lines 66-113, with leader, term, committed index, work state), but it is an ad 
hoc REST endpoint, not something Prometheus scrapes or alerts on.
   
   On the PD side, `StoreNodeService.heartBeat` 
(`hg-pd-core/src/main/java/org/apache/hugegraph/pd/StoreNodeService.java`, 
lines 722-784) persists all of this per store, and 
`TaskScheduleService.patrolStores` 
(`hg-pd-core/src/main/java/org/apache/hugegraph/pd/TaskScheduleService.java`, 
lines 121-129 and 202-229) patrols store liveness every 60 seconds. Yet PD's 
own exporter 
(`hg-pd-service/src/main/java/org/apache/hugegraph/pd/metrics/PDMetrics.java`, 
lines 68-79) only publishes `hg.up`, `hg.graphs`, `hg.stores`, `hg.terms`, 
`hg.partition.terms{id}`, `hg.partitions{graph}`, and `hg.graph.size{graph}`. 
PD knows every store's state and every shard's role and does not export any of 
it either.
   
   ### Proposal
   
   Export a small set of recovery-oriented families on the existing actuator 
endpoint, following the current naming style (dotted micrometer names under the 
`hg.` prefix with the `hg=store` common tag, low-cardinality tags like the 
existing `graph` and `group`):
   
   Store side, registered next to the existing gauges in `StoreMetrics` / a new 
`RecoveryMetrics` source in `MetricsConfig`:
   
   - `hg.partition.state{graph, partition, state}` gauge: 1 when the 
partition's work state equals `state` (`PState_Normal`, `PState_Warn`, 
`PState_Offline`, `PState_Fault`), else 0. Values come straight from 
`Partition.getWorkState()`, already read in `HgMetricService.getStoreMetrics`.
   - `hg.partitions.recovering` gauge: count of local partitions whose work 
state is not `PState_Normal`. Gives dashboards a single "still recovering" 
number per store.
   - `hg.raft.snapshot.installing{group}` gauge: 1 between 
`HgSnapshotHandler.onSnapshotLoad` start and finish.
   - `hg.raft.snapshot.install.bytes{group}` and 
`hg.raft.snapshot.install.bytes.total{group}` gauges: bytes loaded so far 
versus total size of the snapshot data directory, both observable inside 
`onSnapshotLoad`.
   - `hg.raft.log.lag{group}` gauge: leader committed index minus local applied 
index, i.e. the distance still to replay after snapshot install. The local side 
is `PartitionEngine.getCommittedIndex`; the leader side is already shipped to 
PD in `RaftStats`.
   - `hg.raft.leader.count` gauge: size of 
`HgStoreEngine.getLeaderPartition()`. During and after recovery this shows 
leadership draining back to the rebuilt node.
   
   PD side, complementary and cheap because the data is already in 
`StoreNodeService`:
   
   - `hg.store.state{store, state}` gauge: per registered store, 1 for the 
current `StoreState` (`Up`, `Offline`, `Exiting`, `Tombstone`, ...).
   - `hg.store.leader.count{store}` gauge: leaders per store from the shard 
group table, so balance and recovery are visible even while the recovering 
store itself cannot serve its endpoint.
   
   All of these are plain gauges over state the process already holds, so the 
cost is a handful of map lookups per scrape, in line with how `StoreMetrics` 
and `PDMetrics` work today.
   
   ### Context
   
   While testing distributed deployments on Kubernetes, I observed this during 
fault tests that kill a Store pod, drop its volume, and verify the cluster 
heals: grading "healed" today means watching readiness probes and grepping 
logs, which is exactly the gap described above. For reference, the deployment 
tooling used for these tests is the Helm chart contributed in PR #3132 (issue 
#3131); it already scrapes the standard actuator endpoints of Store and PD, so 
these families would light up recovery dashboards and alerts with no deployment 
changes at all. I am happy to work on a PR for the store-side gauges if the 
direction sounds right to the maintainers.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to