Re: How to free used Swap-Space? (from errno=8)
On Wed, Sep 23, 2020 at 12:03:32AM +0300, Konstantin Belousov wrote: ! On Tue, Sep 22, 2020 at 09:11:49PM +0200, Peter wrote: ! > So what happens then is this: ! > ! > $ file scc.e ! > scc.e: ELF 32-bit LSB executable, Intel 80386, version 1 ! > (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, ! > for FreeBSD 9.3 (903504), stripped ! > ! > $ ./scc.e ! > ELF interpreter /libexec/ld-elf.so.1 not found, error 8 ! > Abort trap ! > ! > And this will cost about some (hundred?) kB of swapspace every time it ! > happens. And they do not go away again, neither can the concerned jail ! > do fully die again. ! In what sense it 'costs' ? Well that amount memory gets occupied. Forever, that is, until poweroff/reset. ! Can you show exact sequence of commands and outputs that demostrate your ! point ? What type of filesystem the binaries live on ? Oh, I didn't care. Originally on ZFS. When I tried to reproduce it, most likely on an NFS-4 share, as I didn't bother to put it anywhere special. ! I want to reproduce it locally. Yes that's great! Lets see which info You are lacking. Here we are now on my desktop box (mostly same machine, same configuration, i5-3570, 11.4-p3, amd64). I explicitely removed all the files that do not get installed when /etc/src.conf contains the "WITHOUT_LIB32=", but I have the COMPAT_FREEBSD32 still in the kernel. Now I fetch such an old R9.3/i386 binary from my backups, and drop it into some NFS filesystem: (That binary is only 4kB, I just attach it here, if you wanna try you can straightaway use that one - in normal operation it just converts some words stdin to stdout). admin@disp:510:1/ext/Repos$ dir usr2sys -rwxr-xr-x 1 bin bin4316 Apr 7 2016 usr2sys admin@disp:511:1/ext/Repos$ file usr2sys usr2sys: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 9.3 (903504), stripped admin@disp:513:1/ext/Repos$ mount | grep Repos edge-e:/ext/Repos on /ext/Repos (nfs, nfsv4acls) admin@disp:514:1/ext/Repos$ top | cat Mem: 952M Active, 1687M Inact, 419M Laundry, 4423M Wired, 774M Buf, 348M Free ARC: 1940M Total, 1378M MFU, 172M MRU, 2492K Anon, 48M Header, 340M Other 1134M Compressed, 2749M Uncompressed, 2.43:1 Ratio Swap: 20G Total, 36M Used, 20G Free As we see, this machine has 8 Gig installed and currently about no swap used. Now watch what happens: epos$ ./usr2sys ELF interpreter /libexec/ld-elf.so.1 not found, error 8 Abort trap admin@disp:519:1/ext/Repos$ for i in `seq 1000` > do ./usr2sys > done ELF interpreter /libexec/ld-elf.so.1 not found, error 8 Abort trap ... admin@disp:514:1/ext/Repos$ top | cat Mem: 1010M Active, 1807M Inact, 419M Laundry, 4523M Wired, 774M Buf, 69M Free ARC: 1940M Total, 1383M MFU, 166M MRU, 2503K Anon, 48M Header, 340M Other 1134M Compressed, 2750M Uncompressed, 2.43:1 Ratio Swap: 20G Total, 36M Used, 20G Free The free memory has already disappeared! admin@disp:521:1/ext/Repos$ for i in `seq 5000`; do ./usr2sys ; done ... admin@disp:522:1/ext/Repos$ top | cat Mem: 2154M Active, 78M Inact, 787M Laundry, 4722M Wired, 774M Buf, 89M Free ARC: 1753M Total, 1273M MFU, 97M MRU, 2653K Anon, 39M Header, 340M Other 953M Compressed, 2445M Uncompressed, 2.56:1 Ratio Swap: 20G Total, 358M Used, 20G Free, 1% Inuse Now the swapspace starts filling. Lets see if the placement filesystem makes any difference and go onto UFS: admin@disp:525:1/ext/Repos$ su - Password: root@disp:~ # cp /ext/Repos/usr2sys /var root@disp:~ # dir /var/usr2sys -rwxr-xr-x 1 bin bin 4316 Sep 22 23:55 /var/usr2sys root@disp:~ # mount | grep /var /dev/ada0p5 on /var (ufs, local, soft-updates) admin@disp:527:1/var$ ./usr2sys ELF interpreter /libexec/ld-elf.so.1 not found, error 8 Abort trap admin@disp:521:1/ext/Repos$ for i in `seq 5000`; do ./usr2sys ; done ELF interpreter /libexec/ld-elf.so.1 not found, error 8 Abort trap ... Ahh, that runs a LOT faster now than on the NFS! admin@disp:529:1/var$ top | cat Mem: 1546M Active, 67M Inact, 934M Laundry, 5121M Wired, 774M Buf, 161M Free ARC: 1646M Total, 1159M MFU, 107M MRU, 2686K Anon, 37M Header, 340M Other 849M Compressed, 2257M Uncompressed, 2.66:1 Ratio Swap: 20G Total, 1658M Used, 18G Free, 8% Inuse But memory leakage is similar to worse. admin@disp:530:1/var$ df tmp Filesystem1K-blocks UsedAvail Capacity Mounted on zdesk/var/tmp 24747504 231052 24516452 1%/var/tmp admin@disp:531:1/var$ cp usr2sys tmp admin@disp:532:1/var$ cd tmp admin@disp:533:1/var/tmp$ ./usr2sys ELF interpreter /libexec/ld-elf.so.1 not found, error 8 Abort trap admin@disp:534:1/var/tmp$ for i in `seq 5000`; do ./usr2sys ; done ... You can see this is now a ZFS, and the behaviour is basically the same: Mem: 1497M Active, 5292K Inact, 803M Laundry, 5313M Wired, 774M Buf, 212M Free ARC: 1432M Total, 963M MFU, 105M MRU, 2511K Anon, 21M Header, 341M Other 650M Compressed, 1482M Uncompressed, 2.28:1 Ratio Swap: 20G
Re: How to free used Swap-Space? (from errno=8)
On Wed, Sep 23, 2020 at 12:03:32AM +0300, Konstantin Belousov wrote: > On Tue, Sep 22, 2020 at 09:11:49PM +0200, Peter wrote: > > So what happens then is this: > > > > $ file scc.e > > scc.e: ELF 32-bit LSB executable, Intel 80386, version 1 > > (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, > > for FreeBSD 9.3 (903504), stripped > > > > $ ./scc.e > > ELF interpreter /libexec/ld-elf.so.1 not found, error 8 > > Abort trap > > > > And this will cost about some (hundred?) kB of swapspace every time it > > happens. And they do not go away again, neither can the concerned jail > > do fully die again. > In what sense it 'costs' ? > > Can you show exact sequence of commands and outputs that demostrate your > point ? What type of filesystem the binaries live on ? > > I want to reproduce it locally. I suspect that https://reviews.freebsd.org/D26525 should fix it. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: How to free used Swap-Space? (from errno=8)
On Tue, Sep 22, 2020 at 09:11:49PM +0200, Peter wrote: > So what happens then is this: > > $ file scc.e > scc.e: ELF 32-bit LSB executable, Intel 80386, version 1 > (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, > for FreeBSD 9.3 (903504), stripped > > $ ./scc.e > ELF interpreter /libexec/ld-elf.so.1 not found, error 8 > Abort trap > > And this will cost about some (hundred?) kB of swapspace every time it > happens. And they do not go away again, neither can the concerned jail > do fully die again. In what sense it 'costs' ? Can you show exact sequence of commands and outputs that demostrate your point ? What type of filesystem the binaries live on ? I want to reproduce it locally. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: How to free used Swap-Space? (from errno=8)
I think I can reproduce the problem now. See below. On Tue, Sep 22, 2020 at 02:09:01PM -0400, Mark Johnston wrote: ! On Tue, Sep 22, 2020 at 07:31:07PM +0200, Peter wrote: ! > There is something, and I don't know who owns that: ! > $ vmstat -m | grep shmfd ! > shmfd1314K - 473 64,256,1024,8192 ! > ! > But that doesn't look big either. ! ! That is just the amount of kernel memory used to track a set of objects, ! not the actual object sizes. Unfortunately, in 11 I don't think there's ! any way to enumerate them other than running kgdb and examining the ! shm_dictionary hash table. One of the owners of this is also postgres (maybe among others). ! I think I see a possible problem in i915, though I'm not sure if you'd ! trigger it just by using vt(4). It should be fixed in later FreeBSD ! versions, but is still a problem in 11. Here's a (untested) patch: Thank You, I'll keep that one in store, just in case. But now I found something simpler, while tracking error messages that came into my glance alongside: When patching to 11.4-p3, I had been reluctant to recompile lib32 and install that everywhere, and had kicked it off the systems. And obviousely, I had missed to recompile some of my old self-written binaries and they were still i386 and were called by various scripts. So what happens then is this: $ file scc.e scc.e: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 9.3 (903504), stripped $ ./scc.e ELF interpreter /libexec/ld-elf.so.1 not found, error 8 Abort trap And this will cost about some (hundred?) kB of swapspace every time it happens. And they do not go away again, neither can the concerned jail do fully die again. So, maybe, when removing the lib32 & friends from the system, one must also remove the "options COMPAT_FREEBSD32" from the kernel, so that it might not try to run that binary, and maybe that would avoid the issue. (But then, what if one uses lib32 only in *some* jails? Some evil user in another jail can then bring along an i386 binary and crash the system by bloating the mem.) Anyway, my problem is now solved; as I needed these binaries back in working order anyway. regards, PMc ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: How to free used Swap-Space?
On Tue, Sep 22, 2020 at 07:31:07PM +0200, Peter wrote: > On Tue, Sep 22, 2020 at 12:33:19PM -0400, Mark Johnston wrote: > > ! On Tue, Sep 22, 2020 at 06:08:01PM +0200, Peter wrote: > > ! > my machine should use about 3-4, maybe 5 GB swapspace. Today I found > ! > it suddenly uses 8 GB (which is worryingly near the configured 10G). > ! > > ! > I stopped all the big suckers - nothing found. > ! > I stopped all the jails - no success. > ! > I brought it down to singleuser: it tried to swapoff, but failed. > ! > > ! > I unmounted all filesystems, exported all pools, detached all geli, > ! > and removed most of the netgraphs. Swap is still occupied. > ! > ! > Machine is now running only the init and a shell processes, has > ! > almost no filesystems mounted, has mostly native networks only, and > ! > this still occupies 3 GB of swap which cannot be released. > ! > > ! > What is going on, what is doing this, and how can I get this swapspace > ! > released?? > ! > ! Do you have any shared memory segments lingering? ipcs -a will show > ! SysV shared memory usage. > > I have four small shmem segments from four postgres clusters running. > These should cleanly disappear when the clusters are stopped, and > they are very small. > > Shared Memory: > T ID KEY MODEOWNERGROUPCREATOR CGROUP > NATTCHSEGSZ CPID LPID ATIMEDTIMECTIME > m65536 5432001 --rw--- postgres postgres postgres postgres > 7 48 4793 4793 6:09:34 18:00:31 6:09:34 > m655370 --rw--- postgres postgres postgres postgres > 11 48 6268 6268 6:09:42 10:48:27 6:09:42 > m655380 --rw--- postgres postgres postgres postgres > 5 48 6968 6968 6:09:46 18:28:36 6:09:46 > m655390 --rw--- postgres postgres postgres postgres > 6 48 6992 6992 6:09:47 3:38:34 6:09:47 > > ! For POSIX shared memory, in 11.4 we do not > ! have any good way of listing objects, but "vmstat -m | grep shmfd" will > ! at least show whether any are allocated. > > There is something, and I don't know who owns that: > $ vmstat -m | grep shmfd > shmfd1314K - 473 64,256,1024,8192 > > But that doesn't look big either. That is just the amount of kernel memory used to track a set of objects, not the actual object sizes. Unfortunately, in 11 I don't think there's any way to enumerate them other than running kgdb and examining the shm_dictionary hash table. > Furthermore, this machine is running for quite some time already; it > was running as i386 (with ZFS) until very recently, and I know quite > well what is using much memory: these 3 GB were illegitimate; they > came from nothing I did install. And they are new; this has not > happened before. > > ! If those don't turn anything > ! up then it's possible that there's a swap leak. Do you use any DRM > ! graphics drivers on this system? > > Probably yes. There is no graphics used at all; it just uses "device > vt" in text mode, but it uses i5-3570T CPU (IvyBridge HD2500) graphics > for that, and the driver is "drm2" and "i915drm" from /usr/src/sys (not > those from ports). > Not sure how that would account for 3 GB, unless there is indeed some > leak. I think I see a possible problem in i915, though I'm not sure if you'd trigger it just by using vt(4). It should be fixed in later FreeBSD versions, but is still a problem in 11. Here's a (untested) patch: Index: sys/dev/drm2/i915/i915_gem.c === --- sys/dev/drm2/i915/i915_gem.c(revision 365772) +++ sys/dev/drm2/i915/i915_gem.c(working copy) @@ -1863,6 +1863,8 @@ i915_gem_object_truncate(struct drm_i915_gem_objec vm_obj = obj->base.vm_obj; VM_OBJECT_WLOCK(vm_obj); vm_object_page_remove(vm_obj, 0, 0, false); + if (vm_obj->type == OBJT_SWAP) + swap_pager_freespace(vm_obj, 0, vm_obj->size); VM_OBJECT_WUNLOCK(vm_obj); i915_gem_object_free_mmap_offset(obj); ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: How to free used Swap-Space?
On Tue, Sep 22, 2020 at 12:33:19PM -0400, Mark Johnston wrote: ! On Tue, Sep 22, 2020 at 06:08:01PM +0200, Peter wrote: ! > my machine should use about 3-4, maybe 5 GB swapspace. Today I found ! > it suddenly uses 8 GB (which is worryingly near the configured 10G). ! > ! > I stopped all the big suckers - nothing found. ! > I stopped all the jails - no success. ! > I brought it down to singleuser: it tried to swapoff, but failed. ! > ! > I unmounted all filesystems, exported all pools, detached all geli, ! > and removed most of the netgraphs. Swap is still occupied. ! > ! > Machine is now running only the init and a shell processes, has ! > almost no filesystems mounted, has mostly native networks only, and ! > this still occupies 3 GB of swap which cannot be released. ! > ! > What is going on, what is doing this, and how can I get this swapspace ! > released?? ! ! Do you have any shared memory segments lingering? ipcs -a will show ! SysV shared memory usage. I have four small shmem segments from four postgres clusters running. These should cleanly disappear when the clusters are stopped, and they are very small. Shared Memory: T ID KEY MODEOWNERGROUPCREATOR CGROUP NATTCHSEGSZ CPID LPID ATIMEDTIMECTIME m65536 5432001 --rw--- postgres postgres postgres postgres 7 48 4793 4793 6:09:34 18:00:31 6:09:34 m655370 --rw--- postgres postgres postgres postgres 11 48 6268 6268 6:09:42 10:48:27 6:09:42 m655380 --rw--- postgres postgres postgres postgres 5 48 6968 6968 6:09:46 18:28:36 6:09:46 m655390 --rw--- postgres postgres postgres postgres 6 48 6992 6992 6:09:47 3:38:34 6:09:47 ! For POSIX shared memory, in 11.4 we do not ! have any good way of listing objects, but "vmstat -m | grep shmfd" will ! at least show whether any are allocated. There is something, and I don't know who owns that: $ vmstat -m | grep shmfd shmfd1314K - 473 64,256,1024,8192 But that doesn't look big either. Furthermore, this machine is running for quite some time already; it was running as i386 (with ZFS) until very recently, and I know quite well what is using much memory: these 3 GB were illegitimate; they came from nothing I did install. And they are new; this has not happened before. ! If those don't turn anything ! up then it's possible that there's a swap leak. Do you use any DRM ! graphics drivers on this system? Probably yes. There is no graphics used at all; it just uses "device vt" in text mode, but it uses i5-3570T CPU (IvyBridge HD2500) graphics for that, and the driver is "drm2" and "i915drm" from /usr/src/sys (not those from ports). Not sure how that would account for 3 GB, unless there is indeed some leak. regards, PMc ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: How to free used Swap-Space?
On Tue, Sep 22, 2020 at 06:08:01PM +0200, Peter wrote: > Hi all, > > my machine should use about 3-4, maybe 5 GB swapspace. Today I found > it suddenly uses 8 GB (which is worryingly near the configured 10G). > > I stopped all the big suckers - nothing found. > I stopped all the jails - no success. > I brought it down to singleuser: it tried to swapoff, but failed. > > I unmounted all filesystems, exported all pools, detached all geli, > and removed most of the netgraphs. Swap is still occupied. > > Machine is now running only the init and a shell processes, has > almost no filesystems mounted, has mostly native networks only, and > this still occupies 3 GB of swap which cannot be released. > > What is going on, what is doing this, and how can I get this swapspace > released?? Do you have any shared memory segments lingering? ipcs -a will show SysV shared memory usage. For POSIX shared memory, in 11.4 we do not have any good way of listing objects, but "vmstat -m | grep shmfd" will at least show whether any are allocated. If those don't turn anything up then it's possible that there's a swap leak. Do you use any DRM graphics drivers on this system? > > It is 11.4-RELEASE-p3 amd64. > > > Script started on Mon Sep 21 05:43:20 2020 > root@edge# ps axlww > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 0 0 0 -16 00 752 swapin DLs -291:32.41 [kernel] > 0 1 0 0 20 0 5416 248 wait ILs - 0:00.22 /sbin/init > -- > 0 2 0 0 -16 00 16 ftcl DL- 0:00.00 [ftcleanup] > 0 3 0 0 -16 00 16 crypto_w DL- 0:00.00 [crypto] > 0 4 0 0 -16 00 16 crypto_r DL- 0:00.00 [crypto > returns] > 0 5 0 0 -16 00 32 -DL- 11:41.94 [cam] > 0 6 0 0 -8 00 80 t->zthr_ DL- 13:07.13 [zfskern] > 0 7 0 0 -16 00 16 waiting_ DL- 0:00.00 > [sctp_iterator] > 0 8 0 0 -16 00 16 -DL- 2:05.20 > [rand_harvestq] > 0 9 0 0 -16 00 16 -DL- 0:00.04 [soaiod1] > 010 0 0 155 00 64 -RNL - 17115:06.48 [idle] > 011 0 0 -52 00 352 -WL- 49:05.30 [intr] > 012 0 0 -16 00 64 sleepDL- 16:28.51 [ng_queue] > 013 0 0 -8 00 48 -DL- 23:10.60 [geom] > 014 0 0 -16 00 16 seqstate DL- 0:00.00 [sequencer > 00] > 015 0 0 -68 00 160 -DL- 0:23.64 [usb] > 016 0 0 -16 00 16 -DL- 0:00.04 [soaiod2] > 017 0 0 -16 00 16 -DL- 0:00.04 [soaiod3] > 018 0 0 -16 00 16 -DL- 0:00.04 [soaiod4] > 019 0 0 -16 00 16 idle DL- 0:00.83 > [enc_daemon0] > 020 0 0 -16 00 48 psleep DL- 12:07.72 > [pagedaemon] > 021 0 0 20 00 16 psleep DL- 4:12.41 [vmdaemon] > 022 0 0 155 00 16 pgzero DNL - 0:00.00 [pagezero] > 023 0 0 -16 00 64 psleep DL- 0:23.50 [bufdaemon] > 024 0 0 20 00 16 -DL- 0:04.21 > [bufspacedaemon] > 025 0 0 16 00 16 syncer DL- 0:32.48 [syncer] > 026 0 0 -16 00 16 vlruwt DL- 0:02.31 [vnlru] > 027 0 0 -16 00 16 -DL- 7:11.58 [racctd] > 0 157 0 0 20 00 16 geli:w DL- 0:22.03 [g_eli[0] > ada1p2] > 0 158 0 0 20 00 16 geli:w DL- 0:22.77 [g_eli[1] > ada1p2] > 0 159 0 0 20 00 16 geli:w DL- 0:31.08 [g_eli[2] > ada1p2] > 0 160 0 0 20 00 16 geli:w DL- 0:29.41 [g_eli[3] > ada1p2] > 0 70865 1 0 20 0 7076 3104 wait Ss v0 0:00.21 -sh (sh) > 0 71135 70865 0 20 0 6392 2308 select S+ v0 0:00.00 script > 0 71136 71135 0 23 0 7076 3068 wait Ss0 0:00.00 /bin/sh -i > 0 71142 71136 0 23 0 6928 2584 -R+0 0:00.00 ps axlww > > root@edge# df > Filesystem 512-blocksUsed Avail Capacity Mounted on > /dev/ada3p31936568 860864 92078448%/ > devfs2 2 0 100%/dev > procfs 8 8 0 100%/proc > /dev/ada3p43099192 1184896 166636842%/usr > /dev/ada3p5 5803448112 525808 2%/var > > root@edge# pstat -s > Device 512-blocks UsedAvail Capacity > /dev/ada1p2.eli 10485760 5839232 464652856% > > root@edge# top | cat > last pid: 71147; load averages: 0.19, 0.08, 0.09 up 3+03:21:00 > 05:44:12 > 5 processes:1 running, 4 sleeping > > Mem: 9732K Active, 10M Inact, 882M Laundry, 1920M Wi
How to free used Swap-Space?
Hi all, my machine should use about 3-4, maybe 5 GB swapspace. Today I found it suddenly uses 8 GB (which is worryingly near the configured 10G). I stopped all the big suckers - nothing found. I stopped all the jails - no success. I brought it down to singleuser: it tried to swapoff, but failed. I unmounted all filesystems, exported all pools, detached all geli, and removed most of the netgraphs. Swap is still occupied. Machine is now running only the init and a shell processes, has almost no filesystems mounted, has mostly native networks only, and this still occupies 3 GB of swap which cannot be released. What is going on, what is doing this, and how can I get this swapspace released?? It is 11.4-RELEASE-p3 amd64. Script started on Mon Sep 21 05:43:20 2020 root@edge# ps axlww UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 0 -16 00 752 swapin DLs -291:32.41 [kernel] 0 1 0 0 20 0 5416 248 wait ILs - 0:00.22 /sbin/init -- 0 2 0 0 -16 00 16 ftcl DL- 0:00.00 [ftcleanup] 0 3 0 0 -16 00 16 crypto_w DL- 0:00.00 [crypto] 0 4 0 0 -16 00 16 crypto_r DL- 0:00.00 [crypto returns] 0 5 0 0 -16 00 32 -DL- 11:41.94 [cam] 0 6 0 0 -8 00 80 t->zthr_ DL- 13:07.13 [zfskern] 0 7 0 0 -16 00 16 waiting_ DL- 0:00.00 [sctp_iterator] 0 8 0 0 -16 00 16 -DL- 2:05.20 [rand_harvestq] 0 9 0 0 -16 00 16 -DL- 0:00.04 [soaiod1] 010 0 0 155 00 64 -RNL - 17115:06.48 [idle] 011 0 0 -52 00 352 -WL- 49:05.30 [intr] 012 0 0 -16 00 64 sleepDL- 16:28.51 [ng_queue] 013 0 0 -8 00 48 -DL- 23:10.60 [geom] 014 0 0 -16 00 16 seqstate DL- 0:00.00 [sequencer 00] 015 0 0 -68 00 160 -DL- 0:23.64 [usb] 016 0 0 -16 00 16 -DL- 0:00.04 [soaiod2] 017 0 0 -16 00 16 -DL- 0:00.04 [soaiod3] 018 0 0 -16 00 16 -DL- 0:00.04 [soaiod4] 019 0 0 -16 00 16 idle DL- 0:00.83 [enc_daemon0] 020 0 0 -16 00 48 psleep DL- 12:07.72 [pagedaemon] 021 0 0 20 00 16 psleep DL- 4:12.41 [vmdaemon] 022 0 0 155 00 16 pgzero DNL - 0:00.00 [pagezero] 023 0 0 -16 00 64 psleep DL- 0:23.50 [bufdaemon] 024 0 0 20 00 16 -DL- 0:04.21 [bufspacedaemon] 025 0 0 16 00 16 syncer DL- 0:32.48 [syncer] 026 0 0 -16 00 16 vlruwt DL- 0:02.31 [vnlru] 027 0 0 -16 00 16 -DL- 7:11.58 [racctd] 0 157 0 0 20 00 16 geli:w DL- 0:22.03 [g_eli[0] ada1p2] 0 158 0 0 20 00 16 geli:w DL- 0:22.77 [g_eli[1] ada1p2] 0 159 0 0 20 00 16 geli:w DL- 0:31.08 [g_eli[2] ada1p2] 0 160 0 0 20 00 16 geli:w DL- 0:29.41 [g_eli[3] ada1p2] 0 70865 1 0 20 0 7076 3104 wait Ss v0 0:00.21 -sh (sh) 0 71135 70865 0 20 0 6392 2308 select S+ v0 0:00.00 script 0 71136 71135 0 23 0 7076 3068 wait Ss0 0:00.00 /bin/sh -i 0 71142 71136 0 23 0 6928 2584 -R+0 0:00.00 ps axlww root@edge# df Filesystem 512-blocksUsed Avail Capacity Mounted on /dev/ada3p31936568 860864 92078448%/ devfs2 2 0 100%/dev procfs 8 8 0 100%/proc /dev/ada3p43099192 1184896 166636842%/usr /dev/ada3p5 5803448112 525808 2%/var root@edge# pstat -s Device 512-blocks UsedAvail Capacity /dev/ada1p2.eli 10485760 5839232 464652856% root@edge# top | cat last pid: 71147; load averages: 0.19, 0.08, 0.09 up 3+03:21:0005:44:12 5 processes:1 running, 4 sleeping Mem: 9732K Active, 10M Inact, 882M Laundry, 1920M Wired, 10M Buf, 1023M Free ARC: 335K Total, 16K MFU, 304K MRU, 15K Header 320K Compressed, 2944K Uncompressed, 9.20:1 Ratio Swap: 5120M Total, 2851M Used, 2269M Free, 55% Inuse PID USERNAMETHR PRI NICE SIZERES STATE C TIMEWCPU COMMAND 70865 root 1 200 7076K 3104K wait2 0:00 0.00% sh 71135 root 1 200 6392K 2308K select 1 0:00 0.00% script 71136 root 1 200 7076K 3068K wait2 0:00 0.00% sh 71146 root 1 200 7928K 2980K CPU00 0:00 0.00% top 71147 root 1 200 6300K 2088K piperd 1 0:00 0.00% c