Will do, but I should add that I have done nothing on this box for a couple of months. The day before yesterday I realized that I really needed to backup my laptop, when I went to run my backup script I discovered that I couldn't reach this server. When I went to troubleshoot I couldn't login so I hard rebooted it.
After running fsck in single user mode and letting it fix things I ended up being able to log in which is when I discovered that / was full. Anyway, I'll boot into single user mode later today. I shouldn't be troubleshooting things in the middle of the night. I probably could have retained some more info about my situation if I had waited til the morning to troubleshoot the other night. On Wed, Aug 4, 2021 at 1:14 AM Paul de Weerd <we...@weirdnet.nl> wrote: > On Wed, Aug 04, 2021 at 12:56:57AM -0700, Greg Thomas wrote: > | I take it I'm dealing with filesystem corruption as Ali mentioned > earlier? > > Could be. Boot the system in single user mode or the bsd.rd > installation kernel (at the boot prompt type either 'boot -s' or 'boot > bsd.rd'). Enter the shell and run `fsck /`. > > However, my next guess is that you have some data stored "under" a > mountpoint somewhere. Here's what I mean: > > # mkdir /mnt/test > # du -sh install69.iso > 544M install69.iso > # cp install69.iso /mnt/test > # du -xsh /mnt > 545M /mnt > # vnconfig vnd0 /mnt/test/install69.iso > # mount /dev/vnd0c /mnt/test/ > # du -xsh /mnt > 8.0K /mnt > > Since du can't traverse the hierarchy that the install69.iso image has > been mounted over, it also cannot report on the diskspace used by > files in that hierarchy. > > Again, boot into single user mode (or from bsd.rd) and figure this > out. > > Cheers, > > Paul 'WEiRD' de Weerd > > | On Tue, Aug 3, 2021 at 11:10 PM Otto Moerbeek <o...@drijf.net> wrote: > | > | > On Tue, Aug 03, 2021 at 10:57:42PM -0700, Greg Thomas wrote: > | > > | > > I thought Paul's advice only applies if I was trying to figure it out > | > > before rebooting? I'd already rebooted before sending my first > email. > | > > | > OK, did the free space come back in df after reboot? If so, then it's > | > programs having open files that are unlinked for sure. > | > > | > -Otto > | > > | > > > | > > > | > > > | > > On Tue, Aug 3, 2021 at 10:40 PM Otto Moerbeek <o...@drijf.net> > wrote: > | > > > | > > > On Tue, Aug 03, 2021 at 12:39:54PM -0700, Greg Thomas wrote: > | > > > > | > > > > I'm definitely suffering from filesystem corruption on root. I > had > | > > > > rebooted last night with no change. > | > > > > > | > > > > I have no options for mounting root. > | > > > > > | > > > > grits# cat /etc/fstab > | > > > > 16a27b4b4549ce04.b none swap sw > | > > > > 16a27b4b4549ce04.a / ffs rw 1 1 > | > > > > 16a27b4b4549ce04.k /home ffs rw,nodev,nosuid 1 2 > | > > > > 16a27b4b4549ce04.d /tmp ffs rw,nodev,nosuid 1 2 > | > > > > 16a27b4b4549ce04.f /usr ffs rw,nodev 1 2 > | > > > > 16a27b4b4549ce04.g /usr/X11R6 ffs rw,nodev 1 2 > | > > > > 16a27b4b4549ce04.h /usr/local ffs rw,wxallowed,nodev 1 2 > | > > > > 16a27b4b4549ce04.j /usr/obj ffs rw,nodev,nosuid 1 2 > | > > > > 16a27b4b4549ce04.i /usr/src ffs rw,nodev,nosuid 1 2 > | > > > > 16a27b4b4549ce04.e /var ffs rw,nodev,nosuid 1 2 > | > > > > /dev/sd1c /backup ffs rw,nodev,nosuid 1 2 > | > > > > > | > > > > I need to upgrade so I can do that from scratch. This is my > backup > | > > > server > | > > > > so the configuration is pretty simple. > | > > > > > | > > > > Not sure fsck output helps here? > | > > > > > | > > > > grits# fsck /dev/sd0a > | > > > > ** /dev/rsd0a (NO WRITE) > | > > > > ** Last Mounted on / > | > > > > ** Root file system > | > > > > ** Phase 1 - Check Blocks and Sizes > | > > > > ** Phase 2 - Check Pathnames > | > > > > ** Phase 3 - Check Connectivity > | > > > > ** Phase 4 - Check Reference Counts > | > > > > ** Phase 5 - Check Cyl groups > | > > > > 12852 files, 469195 used, 35516 free (44 frags, 4434 blocks, 0.0% > | > > > > fragmentation) > | > > > > > | > > > > Anyway, I'll reinstall unless someone has more learning > experiences > | > for > | > > > me. > | > > > > > | > > > > And thank you to Paul for giving a quick explanation of the > | > difference > | > > > > between df and du. > | > > > > > | > > > > Thanks all! > | > > > > | > > > fsck looks normal for a mounted filesystem. > | > > > > | > > > but did you try following Paul's advice to find an open file that > has > | > > > no directory entry? That is not corruption, but explains why more > | > > > storage is in use than du shows. > | > > > > | > > > -Otto > | > > > > | > > > > > | > > > > > | > > > > > | > > > > On Tue, Aug 3, 2021 at 11:39 AM Ali Farzanrad < > | > ali_farzan...@riseup.net> > | > > > > wrote: > | > > > > > | > > > > > I also suspected that it is a filesystem corruption. > | > > > > > Do you have `async` mount option on your root? > | > > > > > > | > > > > > Sebastien Marie <sema...@online.fr> wrote: > | > > > > > > On Tue, Aug 03, 2021 at 10:03:44AM +0200, Paul de Weerd > wrote: > | > > > > > > > df shows you how much data you can write to an fs, while du > | > shows > | > > > the > | > > > > > > > disk usage of files it can find. If it can't find a file > | > (because > | > > > > > > > it's been deleted), it won't account for it. But if it's > been > | > > > deleted > | > > > > > > > and still held open by some process, it would still consume > | > disk > | > > > > > > > space. > | > > > > > > > > | > > > > > > > So it looks like a process has a file open on the root > | > filesystem > | > > > that > | > > > > > > > has been deleted. You're looking for a root-owned process > | > that is > | > > > > > > > (probably) long-running. My guess the file is in /dev/ > | > (that's my > | > > > > > > > crystal ball talking though). > | > > > > > > > > | > > > > > > > Easiest way out is generally to reboot - this stops all > | > processes > | > > > > > > > (d0h), dus freeing up all the resources they had tied up, > | > including > | > > > > > > > files that had been deleted from the filesystem. But going > | > through > | > > > > > > > your process list to see if you can spot something that may > | > have > | > > > done > | > > > > > > > this can be a good learning experience. In general, base > | > OpenBSD > | > > > > > > > daemons don't behave this way. > | > > > > > > > | > > > > > > I agree with Paul: you should have a running process which > hold > | > > > > > > descriptor on unlinked file. > | > > > > > > > | > > > > > > fstat(1) could be used to see list of opened files, and > specially > | > > > > > > unlinked files: > | > > > > > > > | > > > > > > INUM The inode number of the file. It will be > followed > | > by an > | > > > > > asterisk > | > > > > > > (‘*’) if the inode is unlinked from disk. > | > > > > > > > | > > > > > > > | > > > > > > $ fstat | grep -F '* -' > | > > > > > > [...] > | > > > > > > semarie chrome 537 25 /tmp 48* -rw------- > | > rwp > | > > > > > 279793 > | > > > > > > [...] > | > > > > > > > | > > > > > > here, chrome (pid 537) has descriptor 25 opened to a file on > /tmp > | > > > > > > inode=48 (unlinked), the file size is 279793 bytes. > | > > > > > > > | > > > > > > -- > | > > > > > > Sebastien Marie > | > > > > > > > | > > > > > > > | > > > > > > | > > > > > > | > > > > | > > > -- > >++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+ > +++++++++++>-]<.>++[<------------>-]<+.--------------.[-] > http://www.weirdnet.nl/ >
