SebastianGruza commented on PR #3164: URL: https://github.com/apache/hugegraph/pull/3164#issuecomment-5647150080
### The race reproduces deterministically On a 3-VM lab (1 PD + 3 Store + 1 Server, replication 1, 25 M edges, partitions of 60–275 MB) the window is not narrow: a full compaction of a partition holding ~250 MB of unflushed data takes 7 s, and a snapshot requested 300 ms after it lands in `state == doing` every time. The compaction is started blocking through `POST :8520/v1/compat?id=`, the snapshot through `GET :8520/test/snapshot`. Script: [`cluster/race_snapshot_compaction.sh`](https://github.com/SebastianGruza/hugegraph-validation/blob/master/cluster/race_snapshot_compaction.sh), store and jraft logs in [`results/issue-3162/`](https://github.com/SebastianGruza/hugegraph-validation/tree/master/results/issue-3162). The only case where the window really disappears is a freshly compacted partition: `compactRange` on 62 MB with nothing to do takes 0.1 s. ### What master does, step by step | step | master `60c8803` | PR head `1c6bc23` | |---|---|---| | snapshot during the compaction | `onSnapshotSave success` in the log, directory `snapshot_50210` with only `__raft_snapshot_meta`, no `data/`, no `should_not_load`; jraft immediately deletes the previous good snapshot and truncates the log prefix to 50209 | `snapshot save failed: compaction in progress`, status EBUSY, no directory at all; the compaction finishes normally and the snapshot after it is valid | | `SIGKILL` the store 1.3 s after the snapshot, restart without wiping | `onSnapshotLoad failed` → `initSnapshotStorage failed` → `Raft 15 is restarting !!!` once, then silence; the partition stays dead | same, only the exception text changes (`snapshot is corrupt, data dir …`) | | what clients see meanwhile | `g.E().count()` = 24 045 646 of 25 000 000, `g.V().count()` = 97 185 of 100 000, HTTP 200, no error on the server side; store REST `/v1/partition/15` returns 500 | same | | `rm -rf snapshot_50210`, restart | `Missing logs in (0, 50209)`: the log prefix is gone, the partition stays dead | same | | `rm -rf raft/00015` (log, meta, snapshot), keep `db/00015`, restart | the partition comes back (100 000 vertices) but **257 001 edges are missing**: HStore runs RocksDB with `setDisableWAL(true)`, the raft log is the WAL, so the unflushed memtable of the killed process is lost together with the log | | ### Two things this says about the PR - **The save side does exactly what is needed.** With the fix no directory without `data/` is created, and more importantly the previous good snapshot is not deleted and the log is not truncated. That is the fix that matters, because once an empty snapshot is committed there is no way back: jraft has removed the previous snapshot and the log prefix, and the local RocksDB has no WAL. - **The load side detects but does not rescue.** The `data/` validation only changes the message; the partition still does not come up and the server keeps answering 200 with partial results. That is not a complaint about the PR, rather a hint that "what to do with a snapshot written before the fix" deserves its own issue: for replication 1 I know no procedure that does not lose data; for replication ≥ 2 a re-install from the leader is probably the natural one. ### Numbers for the 10 s discussion `onSnapshotSave` on a 274 MB partition completes within the same log second (the checkpoint is hard links, CRC64 over the first and last 4 KB of each file), so the "well under a second" assumption holds. The compaction with 250 MB of unflushed data took 7 s, so with `snapshotInterval 1800` a collision is rare, but with a compaction requested after a bulk load it will certainly happen; EBUSY instead of EIO matters there. If it helps, I can repeat the same runs on the next PR head or at replication 3 (where the interesting part is what a follower installing a snapshot in that window receives). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
