Hello,
Today my host had a fault with their cloud node rebooting all servers.
Upon reboot my server stopped at fsck with the following (OCR from
console):
/dev/sdle (81b1b9d368fe02e9.e): UNEXPECTED INCONSISTENCY: RUN fsck_ffs MANUALLY.
THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY:
ffs: 81b1b9d368fe02e9.e (/var)
Automatic file system check failed; help!
Enter pathname of shell or RETURN for sh:
I thought I would save myself the trouble and just restore from the
host provider backup from before the fault, but I got the same fsck
failure. I went back to a day before and got the same. So I did the
fsck_ffs manually:
beastie# fsck_ffs /dev/rsd0e
** /dev/rsd0e
** Last Mounted on /var
** Phase 1 - Check Blocks and Sizes
1875878 DUP I=466625
1875878 DUP I=466626
** Phase 1 - Rescan
For More DUPS
1875878 DUP I=466595
** Phase 2 - Check Pathnames
I=466595 OWNER=bliptown MODE=100600
SIZE=3 MTIME=May 9 03:30 2026
FILE=/db/bliptown/users.db-shm
REMOVE? [Fyn?] y
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
UNREF FILE I=259210
OWNER=root MODE=100444
SIZE=16620 MTIME=Apr 17 04:19 2026
CLEAR? [Fyn?] у
BAD/DUP FILE I=466595
OWNER=bliptown MODE=100600
SIZE=3 MTIME=May 9 03:30 2026
CLEAR? [Fyn?] y
BAD/DUP FILE I=466625
OWNER=bliptown_staging MODE=100600
SIZE=164832 MTIME=May 14 01:30 2026
CLEAR? [Fyn?] y
BAD/DUP FILE I=466626
OWNER=bliptown_staging MODE=100600
SIZE=3 MTIME=May 9 03:30 2026
CLEAR? [Fyn?] у
UNREF FILE I=492549 OWNER=root MODE=100600
SIZE=2535119 MTIME=May
9 00:00 2026
CLEAR? [fyn?] y
** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? [Fyn?] y
SUMMARY INFORMATION BAD
SALVAGE? [fyn?] y
BLK(5) MISSING IN BIT MAPS
SALVAGE? [fyn?] y
9075 files, 71002 used, 1958981 free (245 frags, 244842 blocks, 0.0%
fragmentation)
MARK FILE SYSTEM CLEAN? [Fyn?] y
Everything boots fine and there is no data corruption as far as I can
tell. The files removed were just from the staging SQLite WAL so I'm
unconcerned about those. The other stuff I'm unfamiliar with.
I've got a snapshot of the server in this unclean state and I'd like to
know when and how the corruption occurred, and hopefully avoid it in
the future.
The staging database is copied from the production database every night
with ".backup /var/db/bliptown-staging/users.db" so I suspect this
might have something to do with it?
Does anyone have any ideas on how I might be able to isolate this?
Thanks,
--
Paul W. Rankin
https://rnkn.xyz
