On Thu May 21, 2026 at 4:15 AM UTC, Paul W. Rankin wrote: > Hello, > > Today my host had a fault with their cloud node rebooting all servers. > Upon reboot my server stopped at fsck with the following (OCR from > console): > > /dev/sdle (81b1b9d368fe02e9.e): UNEXPECTED INCONSISTENCY: RUN fsck_ffs > MANUALLY. > THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY: > ffs: 81b1b9d368fe02e9.e (/var) > Automatic file system check failed; help! > Enter pathname of shell or RETURN for sh: > > I thought I would save myself the trouble and just restore from the > host provider backup from before the fault, but I got the same fsck > failure. I went back to a day before and got the same. So I did the > fsck_ffs manually: > > beastie# fsck_ffs /dev/rsd0e > ** /dev/rsd0e > ** Last Mounted on /var > ** Phase 1 - Check Blocks and Sizes > 1875878 DUP I=466625 > 1875878 DUP I=466626 > ** Phase 1 - Rescan > For More DUPS > 1875878 DUP I=466595 > ** Phase 2 - Check Pathnames > I=466595 OWNER=bliptown MODE=100600 > SIZE=3 MTIME=May 9 03:30 2026 > FILE=/db/bliptown/users.db-shm > REMOVE? [Fyn?] y > > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > UNREF FILE I=259210 > OWNER=root MODE=100444 > SIZE=16620 MTIME=Apr 17 04:19 2026 > CLEAR? [Fyn?] у > > BAD/DUP FILE I=466595 > OWNER=bliptown MODE=100600 > SIZE=3 MTIME=May 9 03:30 2026 > CLEAR? [Fyn?] y > > BAD/DUP FILE I=466625 > OWNER=bliptown_staging MODE=100600 > SIZE=164832 MTIME=May 14 01:30 2026 > CLEAR? [Fyn?] y > > BAD/DUP FILE I=466626 > OWNER=bliptown_staging MODE=100600 > SIZE=3 MTIME=May 9 03:30 2026 > CLEAR? [Fyn?] у > > UNREF FILE I=492549 OWNER=root MODE=100600 > SIZE=2535119 MTIME=May > 9 00:00 2026 > CLEAR? [fyn?] y > > ** Phase 5 - Check Cyl groups > FREE BLK COUNT(S) WRONG IN SUPERBLK > SALVAGE? [Fyn?] y > > SUMMARY INFORMATION BAD > SALVAGE? [fyn?] y > > BLK(5) MISSING IN BIT MAPS > SALVAGE? [fyn?] y > > 9075 files, 71002 used, 1958981 free (245 frags, 244842 blocks, 0.0% > fragmentation) > > MARK FILE SYSTEM CLEAN? [Fyn?] y > > Everything boots fine and there is no data corruption as far as I can > tell. The files removed were just from the staging SQLite WAL so I'm > unconcerned about those. The other stuff I'm unfamiliar with. > > I've got a snapshot of the server in this unclean state and I'd like to > know when and how the corruption occurred, and hopefully avoid it in > the future. > > The staging database is copied from the production database every night > with ".backup /var/db/bliptown-staging/users.db" so I suspect this > might have something to do with it? > > Does anyone have any ideas on how I might be able to isolate this? > > Thanks, > > -- > Paul W. Rankin > https://rnkn.xyz
Hi Paul, I'm sorry to hear that. Dicey area to have corruption. I've done a fair bit of crash/pull-the-plug testing with OpenBSD and the results often aren't good. This is more on the paranoid side of things, but I now generally mount all FFS partitions as sync. I am assuming you're not using RAID, which can also make corruption more likely when using 4K sectors (based on my testing.) -Henrich
