SebastianGruza commented on PR #3164: URL: https://github.com/apache/hugegraph/pull/3164#issuecomment-5677119046
Follow-up to the measurement of Sep 12, same lab, now with PD `default-shard-count: 3` (12 partitions × 3 replicas, every partition on every store), 20 M edges, PR head `54d4fe56` vs master `60c8803`. Scripts and logs in the same place ([`results/issue-3162/`](https://github.com/SebastianGruza/hugegraph-validation/tree/master/results/issue-3162), section "Replication 3"). | step | PR head `54d4fe56` | master `60c8803` | |---|---|---| | snapshot 300 ms into the leader's compaction | leader: `snapshot save failed: compaction in progress` (EBUSY), no directory; **both followers run their own `doSnapshot` and save fine** — the snapshot command is a replicated raft task, every replica snapshots its own state machine | leader commits `snapshot_50213` with `__raft_snapshot_meta` and no `data/`; the followers save fine | | `SIGKILL` the leader's store 4 s later, restart without wiping | n/a, nothing to corrupt | `onSnapshotLoad failed` → `Raft 0 is restarting !!!` once; partition 0 on that node is dead (`/v1/partition/0` empty); leadership moved to another node | | the cluster with one dead replica | | `g.E().count()` = 20 000 000, complete: the two healthy replicas serve | | does the dead replica heal itself? | | **no**: its raft node never initialises, so it never asks the leader for a snapshot; the `is restarting` loop ends after one attempt | | operator recovery: `rm -rf raft/00000 db/00000` on the dead node only, restart | | the node joins with `term=0, index=0`, the leader sends `InstallSnapshotRequest` (`lastIncludedLogIndex=50213`), follower `PState_Normal` within a minute, count still 20 000 000 | Conclusion: with three replicas the master bug costs one replica and a manual wipe of one partition on one node, not data; with one replica (the run of Sep 12) it costs the partition and, after the only possible cleanup, the unflushed writes too. The save-side fix in this PR removes the cause in both cases, so after this measurement I see nothing open from my side. The "wipe one partition on one node → InstallSnapshot from the leader" procedure may deserve a sentence in the operations docs, because nothing in the log suggests that this is the way out. One observation outside the scope of this PR, recorded on our side as a separate finding: when the lab's disk filled up during the run, a single-node PD stayed leaderless for good after one failed 5-minute snapshot of its own raft (`Error code = 100` on every call, stores could not register) until it was restarted. If that is of interest to the maintainers, I will file it separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
