On Wed, Sep 23, 2020 at 12:03:32AM +0300, Konstantin Belousov wrote:
! On Tue, Sep 22, 2020 at 09:11:49PM +0200, Peter wrote:
! > So what happens then is this:
! >
! > $ file scc.e
! > scc.e: ELF 32-bit LSB executable, Intel 80386, version 1
! > (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1,
! > for FreeBSD 9.3 (903504), stripped
! >
! > $ ./scc.e
! > ELF interpreter /libexec/ld-elf.so.1 not found, error 8
! > Abort trap
! >
! > And this will cost about some (hundred?) kB of swapspace every time it
! > happens. And they do not go away again, neither can the concerned jail
! > do fully die again.
! In what sense it 'costs' ?
Well that amount memory gets occupied. Forever, that is, until
poweroff/reset.
! Can you show exact sequence of commands and outputs that demostrate your
! point ? What type of filesystem the binaries live on ?
Oh, I didn't care. Originally on ZFS. When I tried to reproduce it,
most likely on an NFS-4 share, as I didn't bother to put it anywhere
special.
! I want to reproduce it locally.
Yes that's great! Lets see which info You are lacking.
Here we are now on my desktop box (mostly same machine, same
configuration, i5-3570, 11.4-p3, amd64).
I explicitely removed all the files that do not get installed
when /etc/src.conf contains the "WITHOUT_LIB32=", but I have the
COMPAT_FREEBSD32 still in the kernel.
Now I fetch such an old R9.3/i386 binary from my backups, and
drop it into some NFS filesystem:
(That binary is only 4kB, I just attach it here, if you wanna try
you can straightaway use that one - in normal operation it just
converts some words stdin to stdout).
admin@disp:510:1/ext/Repos$ dir usr2sys
-rwxr-xr-x 1 bin bin4316 Apr 7 2016 usr2sys
admin@disp:511:1/ext/Repos$ file usr2sys
usr2sys: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD),
dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 9.3 (903504),
stripped
admin@disp:513:1/ext/Repos$ mount | grep Repos
edge-e:/ext/Repos on /ext/Repos (nfs, nfsv4acls)
admin@disp:514:1/ext/Repos$ top | cat
Mem: 952M Active, 1687M Inact, 419M Laundry, 4423M Wired, 774M Buf, 348M Free
ARC: 1940M Total, 1378M MFU, 172M MRU, 2492K Anon, 48M Header, 340M Other
1134M Compressed, 2749M Uncompressed, 2.43:1 Ratio
Swap: 20G Total, 36M Used, 20G Free
As we see, this machine has 8 Gig installed and currently about no swap
used. Now watch what happens:
epos$ ./usr2sys
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
admin@disp:519:1/ext/Repos$ for i in `seq 1000`
> do ./usr2sys
> done
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
...
admin@disp:514:1/ext/Repos$ top | cat
Mem: 1010M Active, 1807M Inact, 419M Laundry, 4523M Wired, 774M Buf, 69M Free
ARC: 1940M Total, 1383M MFU, 166M MRU, 2503K Anon, 48M Header, 340M Other
1134M Compressed, 2750M Uncompressed, 2.43:1 Ratio
Swap: 20G Total, 36M Used, 20G Free
The free memory has already disappeared!
admin@disp:521:1/ext/Repos$ for i in `seq 5000`; do ./usr2sys ; done
...
admin@disp:522:1/ext/Repos$ top | cat
Mem: 2154M Active, 78M Inact, 787M Laundry, 4722M Wired, 774M Buf, 89M Free
ARC: 1753M Total, 1273M MFU, 97M MRU, 2653K Anon, 39M Header, 340M Other
953M Compressed, 2445M Uncompressed, 2.56:1 Ratio
Swap: 20G Total, 358M Used, 20G Free, 1% Inuse
Now the swapspace starts filling.
Lets see if the placement filesystem makes any difference and go onto UFS:
admin@disp:525:1/ext/Repos$ su -
Password:
root@disp:~ # cp /ext/Repos/usr2sys /var
root@disp:~ # dir /var/usr2sys
-rwxr-xr-x 1 bin bin 4316 Sep 22 23:55 /var/usr2sys
root@disp:~ # mount | grep /var
/dev/ada0p5 on /var (ufs, local, soft-updates)
admin@disp:527:1/var$ ./usr2sys
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
admin@disp:521:1/ext/Repos$ for i in `seq 5000`; do ./usr2sys ; done
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
...
Ahh, that runs a LOT faster now than on the NFS!
admin@disp:529:1/var$ top | cat
Mem: 1546M Active, 67M Inact, 934M Laundry, 5121M Wired, 774M Buf, 161M Free
ARC: 1646M Total, 1159M MFU, 107M MRU, 2686K Anon, 37M Header, 340M Other
849M Compressed, 2257M Uncompressed, 2.66:1 Ratio
Swap: 20G Total, 1658M Used, 18G Free, 8% Inuse
But memory leakage is similar to worse.
admin@disp:530:1/var$ df tmp
Filesystem1K-blocks UsedAvail Capacity Mounted on
zdesk/var/tmp 24747504 231052 24516452 1%/var/tmp
admin@disp:531:1/var$ cp usr2sys tmp
admin@disp:532:1/var$ cd tmp
admin@disp:533:1/var/tmp$ ./usr2sys
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
admin@disp:534:1/var/tmp$ for i in `seq 5000`; do ./usr2sys ; done
...
You can see this is now a ZFS, and the behaviour is basically the same:
Mem: 1497M Active, 5292K Inact, 803M Laundry, 5313M Wired, 774M Buf, 212M Free
ARC: 1432M Total, 963M MFU, 105M MRU, 2511K Anon, 21M Header, 341M Other
650M Compressed, 1482M Uncompressed, 2.28:1 Ratio
Swap: 20G