Re: plenty of memory, but system us intensively swapping
Hello,
On 20.11.2018 16:22, Trond Endrestøl wrote:
I know others have created a daemon that observe the ARC and the
amount of wired and free memory, and when these values exceed some
threshold, the daemon will allocate a number of gigabytes, writing
zero to the first byte or word of every page, and then freeing the
allocated memory before going back to sleep.
The ARC will release most of its allocations and the kernel will also
release some but not all of its wired memory, and some user pages are
likely to be thrown onto the swap device, turning the user experience
to a mild nightmare while waiting for applications to be paged back
into memory.
ZFS seems to be the common factor in most, if not all, of these cases.
I created my own and not so sophisticated C program that I run every
now and then:
#include
#include
#include
int main(int argc, char **argv)
{
const size_t pagesize = (size_t)getpagesize();
const size_t gigabyte = 1024ULL * 1024ULL * 1024ULL;
size_t amount, n = 1ULL;
char *p, *offset;
if (argc > 1) {
sscanf(argv[1], "%zu", );
}
amount = n * gigabyte;
if (amount > 0ULL) {
if ( (p = malloc(amount)) != NULL) {
for (offset = p; offset < p + amount; offset += pagesize) {
*offset = '\0';
}
free(p);
}
else {
fprintf(stderr,
"%s:%s:%d: unable to allocate %zu gigabyte%s\n",
argv[0], __FILE__, __LINE__,
n, (n == 1ULL) ? "" : "s");
return 2;
}
}
else {
return 1;
}
return 0;
} // main()
// allocate_gigabytes.c
Jeez, thanks a lot, this stuff is working. Now the system has 8 Gigs of
free memory and stopped swapping.
Well, the next question is addressed to the core team which I suppose
reads this ML eventually - why we don't have something similar as a
watchdog in the base system ? I understand that this solution is
architecturally ugly, but it's now worse not to have any, and it still
works. At least I'm about to run this periodically.
Trond, thanks again.
Eugene.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: plenty of memory, but system us intensively swapping
On Tue, 20 Nov 2018 15:22+0500, Eugene M. Zheganin wrote:
> Hello,
>
> On 20.11.2018 15:12, Trond Endrestøl wrote:
> > On freebsd-hackers the other day,
> > https://lists.freebsd.org/pipermail/freebsd-hackers/2018-November/053575.html,
> > it was suggested to set vm.pageout_update_period=0. This sysctl is at
> > 600 initially.
> >
> > ZFS' ARC needs to be capped, otherwise it will eat most, if not all,
> > of your memory.
> Well, as you can see, ARC ate only half, and the other half is eaten by the
> kernel. So far I suppose that if I will cap the ARC, the kernel will simply
> eat the rest.
I know others have created a daemon that observe the ARC and the
amount of wired and free memory, and when these values exceed some
threshold, the daemon will allocate a number of gigabytes, writing
zero to the first byte or word of every page, and then freeing the
allocated memory before going back to sleep.
The ARC will release most of its allocations and the kernel will also
release some but not all of its wired memory, and some user pages are
likely to be thrown onto the swap device, turning the user experience
to a mild nightmare while waiting for applications to be paged back
into memory.
ZFS seems to be the common factor in most, if not all, of these cases.
I created my own and not so sophisticated C program that I run every
now and then:
#include
#include
#include
int main(int argc, char **argv)
{
const size_t pagesize = (size_t)getpagesize();
const size_t gigabyte = 1024ULL * 1024ULL * 1024ULL;
size_t amount, n = 1ULL;
char *p, *offset;
if (argc > 1) {
sscanf(argv[1], "%zu", );
}
amount = n * gigabyte;
if (amount > 0ULL) {
if ( (p = malloc(amount)) != NULL) {
for (offset = p; offset < p + amount; offset += pagesize) {
*offset = '\0';
}
free(p);
}
else {
fprintf(stderr,
"%s:%s:%d: unable to allocate %zu gigabyte%s\n",
argv[0], __FILE__, __LINE__,
n, (n == 1ULL) ? "" : "s");
return 2;
}
}
else {
return 1;
}
return 0;
} // main()
// allocate_gigabytes.c
--
Trond.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: plenty of memory, but system us intensively swapping
Hello, On 20.11.2018 15:12, Trond Endrestøl wrote: On freebsd-hackers the other day, https://lists.freebsd.org/pipermail/freebsd-hackers/2018-November/053575.html, it was suggested to set vm.pageout_update_period=0. This sysctl is at 600 initially. ZFS' ARC needs to be capped, otherwise it will eat most, if not all, of your memory. Well, as you can see, ARC ate only half, and the other half is eaten by the kernel. So far I suppose that if I will cap the ARC, the kernel will simply eat the rest. Eugene. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: plenty of memory, but system us intensively swapping
On Tue, 20 Nov 2018 14:53+0500, Eugene M. Zheganin wrote: > Hello, > > > I have a recent FreeBSD 11-STABLE which is mainly used as an iSCSI target. The > system has 64G of RAM but is swapping intensively. Yup, about of half of the > memory is used as ZFS ARC (isn't capped in loader.conf), and another half is > eaten by the kernel, but it oly uses only about half of it (thus 25% of the > total amount). > > Could this be tweaked by some sysctl oids (I suppose not, but worth asking). On freebsd-hackers the other day, https://lists.freebsd.org/pipermail/freebsd-hackers/2018-November/053575.html, it was suggested to set vm.pageout_update_period=0. This sysctl is at 600 initially. ZFS' ARC needs to be capped, otherwise it will eat most, if not all, of your memory. > top, vmstat 1 snapshots and zfs-stats -a are listed below. > > > Thanks. > > > [root@san01:nginx/vhost.d]# vmstat 1 > procs memory pagedisks faults cpu > r b w avm fre flt re pi pofr sr da0 da1 in sycs us sy id > 0 0 38 23G 609M 1544 68 118 64 895 839 0 0 3644 2678 649 0 13 > 87 > 0 0 53 23G 601M 1507 185 742 315 1780 33523 651 664 56438 785 476583 0 28 > 72 > 0 0 53 23G 548M 1727 330 809 380 2377 33256 758 763 5 1273 468545 0 > 26 73 > 0 0 53 23G 528M 1702 239 660 305 1347 32335 611 631 59962 1025 490365 0 > 22 78 > 0 0 52 23G 854M 2409 309 693 203 97943 16944 525 515 64309 1570 540533 0 > 29 71 > 3 0 54 23G 1.1G 2756 639 641 149 124049 19531 542 538 64777 1576 553946 0 > 35 65 > 0 0 53 23G 982M 1694 236 680 282 2754 35602 597 603 66540 1385 583687 0 > 28 72 > 0 0 41 23G 867M 1882 223 767 307 1162 34936 682 638 67284 780 568818 0 33 > 67 > 0 0 39 23G 769M 1542 167 673 336 1187 35123 646 610 65925 1176 551623 0 > 23 77 > 2 0 41 23G 700M 3602 535 688 327 2192 37109 622 594 65862 4256 518934 0 > 33 67 > 0 0 54 23G 650M 2957 219 726 464 4838 36464 852 868 65384 4110 558132 1 > 37 62 > 0 0 54 23G 641M 1576 245 730 344 1139 33681 740 679 67216 970 560379 0 31 > 69 > > > [root@san01:nginx/vhost.d]# top > last pid: 55190; load averages: 11.32, 12.15, 10.76 > up 10+16:05:14 14:38:58 > 101 processes: 1 running, 100 sleeping > CPU: 0.2% user, 0.0% nice, 28.9% system, 1.6% interrupt, 69.3% idle > Mem: 85M Active, 1528K Inact, 12K Laundry, 62G Wired, 540M Free > ARC: 31G Total, 19G MFU, 6935M MRU, 2979M Anon, 556M Header, 1046M Other > 25G Compressed, 34G Uncompressed, 1.39:1 Ratio > Swap: 32G Total, 1186M Used, 31G Free, 3% Inuse, 7920K In, 3752K Out > PID USERNAME THR PRI NICE SIZERES STATE C TIMEWCPU COMMAND > 40132 root 131 520 3152M 75876K uwait 14 36:59 6.10% java > 55142 root 1 200 7904K 2728K CPU20 20 0:00 0.72% top > 20026 root 1 200 106M 5676K nanslp 28 1:23 0.60% gstat > 53642 root 1 200 7904K 2896K select 14 0:03 0.58% top > 977 zfsreplica 1 200 30300K 3568K kqread 21 4:00 0.42% uwsgi > 968 zfsreplica 1 200 30300K 2224K swread 11 2:03 0.21% uwsgi > 973 zfsreplica 1 200 30300K 2264K swread 13 12:26 0.13% uwsgi > 53000 www1 200 23376K 1372K kqread 24 0:00 0.05% nginx > 1292 root 1 200 6584K 2040K select 29 0:23 0.04% > blacklistd > 776 zabbix 1 200 12408K 4236K nanslp 26 4:42 0.03% > zabbix_agentd > 1289 root 1 200 67760K 5148K select 13 9:50 0.03% bsnmpd > 777 zabbix 1 200 12408K 1408K select 25 5:06 0.03% > zabbix_agentd > 785 zfsreplica 1 200 27688K 3960K kqread 28 2:04 0.02% uwsgi > 975 zfsreplica 1 200 30300K 464K kqread 18 2:33 0.02% uwsgi > 974 zfsreplica 1 200 30300K 480K kqread 30 3:39 0.02% uwsgi > 965 zfsreplica 1 200 30300K 464K kqread 4 3:23 0.02% uwsgi > 976 zfsreplica 1 200 30300K 464K kqread 14 2:59 0.01% uwsgi > 972 zfsreplica 1 200 30300K 464K kqread 10 2:57 0.01% uwsgi > 963 zfsreplica 1 200 30300K 460K kqread 3 2:45 0.01% uwsgi > 971 zfsreplica 1 200 30300K 464K kqread 13 3:16 0.01% uwsgi > 69644 emz1 200 13148K 4596K select 24 0:05 0.01% sshd > 18203 vryabov1 200 13148K 4624K select 9 0:02 0.01% sshd > 636 root 1 200 6412K 1884K select 17 4:10 0.01% syslogd > 51266 emz1 200 13148K 4576K select 5 0:00 0.01% sshd > 964 zfsreplica 1 200 30300K 460K kqread 18 11:02 0.01% uwsgi > 962 zfsreplica 1 200 30300K 460K kqread 28 6:56 0.01% uwsgi > 969 zfsreplica 1 200 30300K 464K kqread 12 2:07 0.01% uwsgi > 967 zfsreplica 1 200 30300K 464K kqread 27 5:18 0.01% uwsgi > 970 zfsreplica 1 200 30300K 464K kqread 0 4:25 0.01% uwsgi > 966
plenty of memory, but system us intensively swapping
Hello, I have a recent FreeBSD 11-STABLE which is mainly used as an iSCSI target. The system has 64G of RAM but is swapping intensively. Yup, about of half of the memory is used as ZFS ARC (isn't capped in loader.conf), and another half is eaten by the kernel, but it oly uses only about half of it (thus 25% of the total amount). Could this be tweaked by some sysctl oids (I suppose not, but worth asking). top, vmstat 1 snapshots and zfs-stats -a are listed below. Thanks. [root@san01:nginx/vhost.d]# vmstat 1 procs memory pagedisks faults cpu r b w avm fre flt re pi pofr sr da0 da1 in sycs us sy id 0 0 38 23G 609M 1544 68 118 64 895 839 0 0 3644 2678 649 0 13 87 0 0 53 23G 601M 1507 185 742 315 1780 33523 651 664 56438 785 476583 0 28 72 0 0 53 23G 548M 1727 330 809 380 2377 33256 758 763 5 1273 468545 0 26 73 0 0 53 23G 528M 1702 239 660 305 1347 32335 611 631 59962 1025 490365 0 22 78 0 0 52 23G 854M 2409 309 693 203 97943 16944 525 515 64309 1570 540533 0 29 71 3 0 54 23G 1.1G 2756 639 641 149 124049 19531 542 538 64777 1576 553946 0 35 65 0 0 53 23G 982M 1694 236 680 282 2754 35602 597 603 66540 1385 583687 0 28 72 0 0 41 23G 867M 1882 223 767 307 1162 34936 682 638 67284 780 568818 0 33 67 0 0 39 23G 769M 1542 167 673 336 1187 35123 646 610 65925 1176 551623 0 23 77 2 0 41 23G 700M 3602 535 688 327 2192 37109 622 594 65862 4256 518934 0 33 67 0 0 54 23G 650M 2957 219 726 464 4838 36464 852 868 65384 4110 558132 1 37 62 0 0 54 23G 641M 1576 245 730 344 1139 33681 740 679 67216 970 560379 0 31 69 [root@san01:nginx/vhost.d]# top last pid: 55190; load averages: 11.32, 12.15, 10.76 up 10+16:05:14 14:38:58 101 processes: 1 running, 100 sleeping CPU: 0.2% user, 0.0% nice, 28.9% system, 1.6% interrupt, 69.3% idle Mem: 85M Active, 1528K Inact, 12K Laundry, 62G Wired, 540M Free ARC: 31G Total, 19G MFU, 6935M MRU, 2979M Anon, 556M Header, 1046M Other 25G Compressed, 34G Uncompressed, 1.39:1 Ratio Swap: 32G Total, 1186M Used, 31G Free, 3% Inuse, 7920K In, 3752K Out PID USERNAME THR PRI NICE SIZERES STATE C TIMEWCPU COMMAND 40132 root 131 520 3152M 75876K uwait 14 36:59 6.10% java 55142 root 1 200 7904K 2728K CPU20 20 0:00 0.72% top 20026 root 1 200 106M 5676K nanslp 28 1:23 0.60% gstat 53642 root 1 200 7904K 2896K select 14 0:03 0.58% top 977 zfsreplica 1 200 30300K 3568K kqread 21 4:00 0.42% uwsgi 968 zfsreplica 1 200 30300K 2224K swread 11 2:03 0.21% uwsgi 973 zfsreplica 1 200 30300K 2264K swread 13 12:26 0.13% uwsgi 53000 www1 200 23376K 1372K kqread 24 0:00 0.05% nginx 1292 root 1 200 6584K 2040K select 29 0:23 0.04% blacklistd 776 zabbix 1 200 12408K 4236K nanslp 26 4:42 0.03% zabbix_agentd 1289 root 1 200 67760K 5148K select 13 9:50 0.03% bsnmpd 777 zabbix 1 200 12408K 1408K select 25 5:06 0.03% zabbix_agentd 785 zfsreplica 1 200 27688K 3960K kqread 28 2:04 0.02% uwsgi 975 zfsreplica 1 200 30300K 464K kqread 18 2:33 0.02% uwsgi 974 zfsreplica 1 200 30300K 480K kqread 30 3:39 0.02% uwsgi 965 zfsreplica 1 200 30300K 464K kqread 4 3:23 0.02% uwsgi 976 zfsreplica 1 200 30300K 464K kqread 14 2:59 0.01% uwsgi 972 zfsreplica 1 200 30300K 464K kqread 10 2:57 0.01% uwsgi 963 zfsreplica 1 200 30300K 460K kqread 3 2:45 0.01% uwsgi 971 zfsreplica 1 200 30300K 464K kqread 13 3:16 0.01% uwsgi 69644 emz1 200 13148K 4596K select 24 0:05 0.01% sshd 18203 vryabov1 200 13148K 4624K select 9 0:02 0.01% sshd 636 root 1 200 6412K 1884K select 17 4:10 0.01% syslogd 51266 emz1 200 13148K 4576K select 5 0:00 0.01% sshd 964 zfsreplica 1 200 30300K 460K kqread 18 11:02 0.01% uwsgi 962 zfsreplica 1 200 30300K 460K kqread 28 6:56 0.01% uwsgi 969 zfsreplica 1 200 30300K 464K kqread 12 2:07 0.01% uwsgi 967 zfsreplica 1 200 30300K 464K kqread 27 5:18 0.01% uwsgi 970 zfsreplica 1 200 30300K 464K kqread 0 4:25 0.01% uwsgi 966 zfsreplica 1 220 30300K 468K kqread 14 4:29 0.01% uwsgi 53001 www1 200 23376K 1256K kqread 10 0:00 0.01% nginx 791 zfsreplica 1 200 27664K 4244K kqread 17 1:34 0.01% uwsgi 52431 root 1 200 17132K 4492K select 21 0:00 0.01% mc 70013 root 1 200 17132K 4492K select 4 0:03 0.01% mc 870 root 1 200 12448K 12544K select 19
