SebastianGruza opened a new issue, #3222:
URL: https://github.com/apache/hugegraph/issues/3222

   ### Bug Type (问题类型)
   
   others (availability)
   
   ### Before submit
   
   - [X] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 
[FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have 
confirmed and searched that there are no similar problems in the historical 
issue and documents)
   
   ### Environment (环境信息)
   
   - PD and Store `:latest` of 2026-09-18 (master after #3210), jraft-core 
1.3.13; chart of #3218 / hugegraph/hugegraph#221, `values-single.yaml` (1 PD + 
1 Store + 1 Server) on a 3-node k3s
   - PD data on its own 512 MB volume (loop-backed ext4 as a hostPath PV), so 
ENOSPC hits PD only
   - The same outcome was seen on 2026-09-15 on VMs with PD 1.7.0 and a root 
disk at 98 %
   
   ### Expected & Actual behavior (期望与实际表现)
   
   Expected: with a full disk PD stops accepting writes and tells the probes 
so; once the disk is freed it resumes on its own, or at least `/v1/health` 
fails so an orchestrator restarts it.
   
   Actual (two passes, `results/pd-disk-full-single` in the validation repo):
   
   | t | event | `/v1/ready` | `/v1/health` | clients |
   |---|---|---|---|---|
   | 0 | volume 49 % used (raft log preallocation) | 200, `STATE_LEADER` | 200 
| schema writes 202, reads 200 |
   | +60 s | `dd` fills the volume to 100 % | still 200 for ~1 min | 200 | 
unchanged |
   | +116 s | the periodic raft snapshot fails on ENOSPC: `RaftException: 
ERROR_TYPE_SNAPSHOT` from `SnapshotExecutorImpl.doSnapshot` → 
`RaftStateMachine.onError` → `Raft lost leader` | 503, 
`{"ready":false,"state":"STATE_FOLLOWER","isLeader":false}` | 200 | every 
gRPC/REST call: `IllegalStateException: Leader is not ready` (63 in 5 min); 
schema writes through the Server time out; reads through the Server still 200 |
   | +241 s | filler removed, volume back to 49 % | **503 for the next 180 s**, 
not a single election attempt in the log | 200 | writes still fail |
   | +422 s | PD pod restarted | 200, `STATE_LEADER` after ~30 s | 200 | writes 
202; oracle over the whole run: 0 mismatches, 0 acknowledged writes lost |
   
   ### Analysis
   
   - The trigger is a failed **periodic** snapshot 
(`NodeImpl.handleSnapshotTimeout`), not a log write, so PD looks healthy for 
about a minute after the disk fills and drops at a random moment relative to 
the fault.
   - After `onError`, jraft 1.3.13 steps the node down and never starts an 
election again (the node stays in its error state until the process restarts); 
in a one-member group nobody else can become leader, so PD stays leaderless for 
good, even after the cause is gone. With 3 PD the other two elect a leader and 
the symptom is one PD permanently out of the quorum.
   - `RaftStateMachine.onError` sets `ProbeView(STATE_ERROR, false)`, but 
`onLeaderStop`, which jraft calls a moment later, unconditionally overwrites it 
with `ProbeView(STATE_FOLLOWER, false)`; `/v1/ready` therefore shows "a 
follower without a leader", not an error. `/v1/health` is `return ""`, 200 as 
long as Jetty is up, so no liveness probe ever restarts such a PD.
   
   ### Proposal
   
   1. `onLeaderStop` / `onStopFollowing` must not overwrite `STATE_ERROR` (or 
`getRaftStatus()` asks `node.getNodeState()`), so that `/v1/ready` tells the 
truth.
   2. `/v1/health` answers 503 when the raft node is in `STATE_ERROR`: that 
state is terminal until restart, so it is the right liveness signal; on a plain 
loss of leader (3 PD) health stays 200 as today.
   3. Longer term: after a snapshot error PD could retry the snapshot and, once 
the cause is gone, re-initialise its raft node without a process restart; at 
minimum an upgrade note for single-node installations.
   
   Repro: `cluster/k3s_pd_disk_full.py` plus `pd_fill_capture.sh` in 
https://github.com/SebastianGruza/hugegraph-validation; the PD log of pass 2 in 
`results/pd-disk-full-single/pass2/`.
   
   ### Vertex/Edge example (问题点 / 边数据举例)
   
   _No response_
   
   ### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
   
   _No response_
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to