SebastianGruza opened a new issue, #3222: URL: https://github.com/apache/hugegraph/issues/3222
### Bug Type (问题类型) others (availability) ### Before submit - [X] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 [FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents) ### Environment (环境信息) - PD and Store `:latest` of 2026-09-18 (master after #3210), jraft-core 1.3.13; chart of #3218 / hugegraph/hugegraph#221, `values-single.yaml` (1 PD + 1 Store + 1 Server) on a 3-node k3s - PD data on its own 512 MB volume (loop-backed ext4 as a hostPath PV), so ENOSPC hits PD only - The same outcome was seen on 2026-09-15 on VMs with PD 1.7.0 and a root disk at 98 % ### Expected & Actual behavior (期望与实际表现) Expected: with a full disk PD stops accepting writes and tells the probes so; once the disk is freed it resumes on its own, or at least `/v1/health` fails so an orchestrator restarts it. Actual (two passes, `results/pd-disk-full-single` in the validation repo): | t | event | `/v1/ready` | `/v1/health` | clients | |---|---|---|---|---| | 0 | volume 49 % used (raft log preallocation) | 200, `STATE_LEADER` | 200 | schema writes 202, reads 200 | | +60 s | `dd` fills the volume to 100 % | still 200 for ~1 min | 200 | unchanged | | +116 s | the periodic raft snapshot fails on ENOSPC: `RaftException: ERROR_TYPE_SNAPSHOT` from `SnapshotExecutorImpl.doSnapshot` → `RaftStateMachine.onError` → `Raft lost leader` | 503, `{"ready":false,"state":"STATE_FOLLOWER","isLeader":false}` | 200 | every gRPC/REST call: `IllegalStateException: Leader is not ready` (63 in 5 min); schema writes through the Server time out; reads through the Server still 200 | | +241 s | filler removed, volume back to 49 % | **503 for the next 180 s**, not a single election attempt in the log | 200 | writes still fail | | +422 s | PD pod restarted | 200, `STATE_LEADER` after ~30 s | 200 | writes 202; oracle over the whole run: 0 mismatches, 0 acknowledged writes lost | ### Analysis - The trigger is a failed **periodic** snapshot (`NodeImpl.handleSnapshotTimeout`), not a log write, so PD looks healthy for about a minute after the disk fills and drops at a random moment relative to the fault. - After `onError`, jraft 1.3.13 steps the node down and never starts an election again (the node stays in its error state until the process restarts); in a one-member group nobody else can become leader, so PD stays leaderless for good, even after the cause is gone. With 3 PD the other two elect a leader and the symptom is one PD permanently out of the quorum. - `RaftStateMachine.onError` sets `ProbeView(STATE_ERROR, false)`, but `onLeaderStop`, which jraft calls a moment later, unconditionally overwrites it with `ProbeView(STATE_FOLLOWER, false)`; `/v1/ready` therefore shows "a follower without a leader", not an error. `/v1/health` is `return ""`, 200 as long as Jetty is up, so no liveness probe ever restarts such a PD. ### Proposal 1. `onLeaderStop` / `onStopFollowing` must not overwrite `STATE_ERROR` (or `getRaftStatus()` asks `node.getNodeState()`), so that `/v1/ready` tells the truth. 2. `/v1/health` answers 503 when the raft node is in `STATE_ERROR`: that state is terminal until restart, so it is the right liveness signal; on a plain loss of leader (3 PD) health stays 200 as today. 3. Longer term: after a snapshot error PD could retry the snapshot and, once the cause is gone, re-initialise its raft node without a process restart; at minimum an upgrade note for single-node installations. Repro: `cluster/k3s_pd_disk_full.py` plus `pd_fill_capture.sh` in https://github.com/SebastianGruza/hugegraph-validation; the PD log of pass 2 in `results/pd-disk-full-single/pass2/`. ### Vertex/Edge example (问题点 / 边数据举例) _No response_ ### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构) _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
