Re: lightly loaded system eats swap space
Hi, On Mon, 18 Jun 2018 13:27:23 +0100 tech-lists wrote: > On 18/06/2018 09:08, Erich Dollansky wrote: > > > > On Sun, 17 Jun 2018 23:19:02 +0100 > > tech-lists wrote: > > > >> freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, > >> Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse > > > > this might not be related but I noticed that your swap space is > > small compared to RAM size. I noticed on a much smaller Raspberry > > Pi, that it runs into trouble when there is no swap even there is > > enough RAM available. Is it easily possible for you to add some GB > > of swap space and let the machine run then? > > > > How much swap do the other machines have? > > Yes, the machine with the problem uses the default 4GB swap. That's > all the swap it has. The machine without issue has a swapfile > installed on a SSD in addition to the default 4GB swap. > > problematic machine: > Device 512-blocks UsedAvail Capacity > /dev/ada0p38388608 3.3G 714M83% > > machine without a problem, it has swapfile installed: > Device 512-blocks UsedAvail Capacity > /dev/ada0s1b 8262248 1.7G 2.2G44% > /dev/md0 65536000 1.9G 29G 6% > Total 73798248 3.7G 32G10% > > I added the swapfile a long time ago on this machine due to the same > issue. so, the same effect as on a small Raspberry. It seems that you also use a memory disk for swap. Mine is backed by a file via NFS. > > But my problem isn't so much an out of swapspace problem; all this > is, is a symptom. My problem is "why is it swapping out at all on a > 128GB system and why is what's swapped out not being swapped back in > again". > I wondered even on the small Raspberry about this. The Raspberries come with 1GB of RAM. Running just a compilation should never be the problem but sometimes it is. A very long time ago - and not on FreeBSD but maybe on a real BSD - I worked with a system that swapped pages out just to bring it back as one contiguous block. This made a difference those days. I do not know if the code made it out of the university I was working at. I just imagine now that the code made it out and is still in use with the opposite effect. Erich ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Ryzen issues on FreeBSD ? (with sort of workaround)
On 06/18/2018 09:34, Pete French wrote: > Preseumably in the slightly longer term these workarounds go into the > actual kernel if it detects Ryzen ? Yes, Kostik said he would code this into the kernel after he gets enough feedback. Eric ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: lightly loaded system eats swap space
On Mon, Jun 18, 2018 at 5:27 AM, tech-lists wrote: > On 18/06/2018 09:08, Erich Dollansky wrote: > >> Hi, >> >> On Sun, 17 Jun 2018 23:19:02 +0100 >> tech-lists wrote: >> >> freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, >>> Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse >>> >> >> this might not be related but I noticed that your swap space is small >> compared to RAM size. I noticed on a much smaller Raspberry Pi, that it >> runs into trouble when there is no swap even there is enough RAM >> available. Is it easily possible for you to add some GB of swap space >> and let the machine run then? >> >> How much swap do the other machines have? >> > > Hi, > > Yes, the machine with the problem uses the default 4GB swap. That's all > the swap it has. The machine without issue has a swapfile installed on a > SSD in addition to the default 4GB swap. > > problematic machine: > Device 512-blocks UsedAvail Capacity > /dev/ada0p38388608 3.3G 714M83% > > machine without a problem, it has swapfile installed: > Device 512-blocks UsedAvail Capacity > /dev/ada0s1b 8262248 1.7G 2.2G44% > /dev/md0 65536000 1.9G 29G 6% > Total 73798248 3.7G 32G10% > > I added the swapfile a long time ago on this machine due to the same issue. > > But my problem isn't so much an out of swapspace problem; all this is, is > a symptom. My problem is "why is it swapping out at all on a 128GB system > and why is what's swapped out not being swapped back in again". > > thanks, > -- > J. Small correction. Your problem is "why is it swapping out at all on a 128GB system ." Once pages are swapped out, they are never swapped back in until/unless they are needed. There is no reason to waste time/disk activity to swap pages back into memory unless they are required. RAM is always more valuable than swap. Ir is easy to write a process that eats up a large amount of memory, then goes idle without freeing it. The memory will get pages out, fill swap, and, unless the process terminates or becomes active, will consume up a great deal of swap space "forever". Firefox is a good example of this. I have to restart it every day or two and occasionally will run out of swap which results in a nearly deadlocked system. It can take many minutes to just kill firefox. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkober...@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Ryzen issues on FreeBSD ? (with sort of workaround)
On 18/06/2018 14:59, Mike Tancsa wrote: FYI, both my Epyc and Ryzen system have been running 2+ days with the tests that would normally hard lock the system in 5-60 min. The combo of Microcode updates and system settings Thats for the update - I turned all the default motherboard settings back on when I got into work this morning, and it has been running fine so far. WIll run like this all week. https://lists.freebsd.org/pipermail/freebsd-current/2018-June/069799.html seem to have fixed the last issue I was seeing Yes, those are the chnages I have applied. I went through that list over the weekend cross-referncing them to the errata document, and there are a couple of things in there which made me think 'aha!' - particularly the MWWAIT causing lockups on SMT, as it was disbling SMT that made it stable for me. I also have access to a cpuple of (virtualised) Epyc machines which I am going to run up later in the week and try the workarounds there. But it does look very much like its fixed, which is excellent. Preseumably in the slightly longer term these workarounds go into the actual kernel if it detects Ryzen ? -pete. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Ryzen issues on FreeBSD ? (with sort of workaround)
On 6/14/2018 6:30 AM, Pete French wrote: >> Check out this thread on current. I re-ran the tests I did in Feb to >> lockup the Ryzen box, and they are ok now with the latest micro code >> updates from AMD. I will let the tests run a good 48hrs, but in the >> past it would only take 5-10 min to cause a hard lockup >> >> https://lists.freebsd.org/pipermail/freebsd-current/2018-June/069799.html > > Ah, excellent, tanksf for the pinter. I was already running the latest > microcode, but have also now flashed my BIOS and applied the shell > script fixes from that thread. Will try running with SMT re-enabled > next week, as I need the machine not to lockup the next few days > when I am not in the office, but it looks very promising. FYI, both my Epyc and Ryzen system have been running 2+ days with the tests that would normally hard lock the system in 5-60 min. The combo of Microcode updates and system settings https://lists.freebsd.org/pipermail/freebsd-current/2018-June/069799.html seem to have fixed the last issue I was seeing ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 x203 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: lightly loaded system eats swap space
On 18/06/2018 13:23, David Fullard wrote: I've noticed you've got a rather large ZFS ARC. You could try limiting the ZFS max ARC size by setting the vfs.zfs.arc_max sysctl. I'll try this as soon as I can. thanks, -- J. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: lightly loaded system eats swap space
On Mon, Jun 18, 2018 at 7:27 AM, tech-lists wrote: > On 18/06/2018 09:08, Erich Dollansky wrote: > >> Hi, >> >> On Sun, 17 Jun 2018 23:19:02 +0100 >> tech-lists wrote: >> >> freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, >>> Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse >>> >> >> this might not be related but I noticed that your swap space is small >> compared to RAM size. I noticed on a much smaller Raspberry Pi, that it >> runs into trouble when there is no swap even there is enough RAM >> available. Is it easily possible for you to add some GB of swap space >> and let the machine run then? >> >> How much swap do the other machines have? >> > > Hi, > > Yes, the machine with the problem uses the default 4GB swap. That's all > the swap it has. The machine without issue has a swapfile installed on a > SSD in addition to the default 4GB swap. > > problematic machine: > Device 512-blocks UsedAvail Capacity > /dev/ada0p38388608 3.3G 714M83% > > machine without a problem, it has swapfile installed: > Device 512-blocks UsedAvail Capacity > /dev/ada0s1b 8262248 1.7G 2.2G44% > /dev/md0 65536000 1.9G 29G 6% > Total 73798248 3.7G 32G10% > > I added the swapfile a long time ago on this machine due to the same issue. > > But my problem isn't so much an out of swapspace problem; all this is, is > a symptom. My problem is "why is it swapping out at all on a 128GB system > and why is what's swapped out not being swapped back in again". > What is the output of sysctl vm.overcommit? If this system is intended on being a VM host, then why don't you limit ARC to something reasonable like Total Mem - Projected VM Mem - Overhead = Ideal ARC . -- Adam ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: lightly loaded system eats swap space
On 18/06/2018 09:08, Erich Dollansky wrote: Hi, On Sun, 17 Jun 2018 23:19:02 +0100 tech-lists wrote: freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse this might not be related but I noticed that your swap space is small compared to RAM size. I noticed on a much smaller Raspberry Pi, that it runs into trouble when there is no swap even there is enough RAM available. Is it easily possible for you to add some GB of swap space and let the machine run then? How much swap do the other machines have? Hi, Yes, the machine with the problem uses the default 4GB swap. That's all the swap it has. The machine without issue has a swapfile installed on a SSD in addition to the default 4GB swap. problematic machine: Device 512-blocks UsedAvail Capacity /dev/ada0p38388608 3.3G 714M83% machine without a problem, it has swapfile installed: Device 512-blocks UsedAvail Capacity /dev/ada0s1b 8262248 1.7G 2.2G44% /dev/md0 65536000 1.9G 29G 6% Total 73798248 3.7G 32G10% I added the swapfile a long time ago on this machine due to the same issue. But my problem isn't so much an out of swapspace problem; all this is, is a symptom. My problem is "why is it swapping out at all on a 128GB system and why is what's swapped out not being swapped back in again". thanks, -- J. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: lightly loaded system eats swap space
I've noticed you've got a rather large ZFS ARC. You could try limiting the ZFS max ARC size by setting the vfs.zfs.arc_max sysctl. On Sun, Jun 17, 2018, at 6:19 PM, tech-lists wrote: > Hello list, > > context is (server) > freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, > E5-2630 @2.3GHz, generic kernel. > > There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also > freebsd-11-stable) > > There have been no special options for zfs configuration on the server, > apart from several datasets having the compressed property set (lz4). > > The server runs nothing else really apart from sshd and it uses ntpd to > sync local time. > > How come such a lightly loaded server with plenty of resources is eating > up swap? If I run two bhyve instances, i.e. two of the same size as > indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace > errors in the daily logs: > > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > > Here's top, with one bhyve instance running: > > last pid: 49494; load averages: 0.12, 0.13, 0.88 > > > up 29+11:36:06 22:52:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle > Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, > 4123M Free > ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other > 103G Compressed, 104G Uncompressed, 1.00:1 Ratio > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse > >PID USERNAMETHR PRI NICE SIZERES STATE C TIMEWCPU > COMMAND > 49491 root 1 40 16444K 12024K select 9 0:12 6.49% ssh > 32868 root 12 200 9241M 4038M kqread 2 23.2H 1.30% bhyve > 49490 root 1 200 10812K 6192K sbwait 5 0:02 0.88% sftp > > From the looks of it, a huge amount of ram is wired. Why is that, and > how would I debug it? > > A server of similar spec which is running freebsd-current with seven > bhyve instances doesn't have this issue: > > last pid: 41904; load averages: 0.26, 0.19, 0.15 > > > up 17+01:06:11 23:14:13 > 27 processes: 1 running, 26 sleeping > CPU: 0.1% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.6% idle > Mem: 17G Active, 6951M Inact, 41G Laundry, 59G Wired, 1573M Buf, 1315M Free > ARC: 53G Total, 700M MFU, 52G MRU, 512K Anon, 182M Header, 958K Other > 53G Compressed, 69G Uncompressed, 1.30:1 Ratio, 122M Overhead > Swap: 35G Total, 2163M Used, 33G Free, 6% Inuse > > thanks, > -- > J. > ___ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: lightly loaded system eats swap space
On 18/06/2018 00:04, Adam wrote: Based upon the output neither ram nor swap seems like similar spec so I wonder if you could say what you mean by that. server with the problem: last pid: 62387; load averages: 0.07, 0.10, 0.08 up 30+01:30:01 12:46:40 48 processes: 1 running, 47 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 5404K Active, 1218M Inact, 4927M Laundry, 117G Wired, 1392M Buf, 2116M Free ARC: 109G Total, 2982M MFU, 105G MRU, 288K Anon, 385M Header, 638M Other 104G Compressed, 105G Uncompressed, 1.01:1 Ratio Swap: 4096M Total, 3383M Used, 713M Free, 82% Inuse This server runs 1 bhyve instance using 16GB. It has 128GB RAM. server without a problem: last pid: 84491; load averages: 0.03, 0.02, 0.02 up 17+14:39:31 12:47:33 27 processes: 1 running, 26 sleeping CPU: % user, % nice, % system, % interrupt, % idle Mem: 2915M Active, 25G Inact, 9947M Laundry, 28G Wired, 1572M Buf, 59G Free ARC: 22G Total, 3379M MFU, 19G MRU, 781K Anon, 64M Header, 974K Other 22G Compressed, 29G Uncompressed, 1.33:1 Ratio, 121M Overhead Swap: 35G Total, 3747M Used, 32G Free, 10% Inuse This server runs 10 bhyve instances of various RAM sizes, from 4GB to 32GB. It also has 128GB RAM installed. This is what I mean by "similar spec", I mean in terms of RAM. This server doesn't usually show swap as being in use, but I can actually account for this, as there is memory overcommitment here. My point is that 'server with problem' should not show a resources issue with just one bhyve instance using 16GB RAM but apparently it is, and I don't know why. I am starting the VMs by hand, without any third-party application, and I'm not wiring memory on the command line. thanks, -- J. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: lightly loaded system eats swap space
Hi, On Sun, 17 Jun 2018 23:19:02 +0100 tech-lists wrote: > freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse this might not be related but I noticed that your swap space is small compared to RAM size. I noticed on a much smaller Raspberry Pi, that it runs into trouble when there is no swap even there is enough RAM available. Is it easily possible for you to add some GB of swap space and let the machine run then? How much swap do the other machines have? Erich ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: syslogd became silent between 11.2-PRERELEASE r334874 and r335282
Hi Michael, 2018-06-17 22:04 GMT+02:00 Michael Grimm : > Thanks, your patch fixed it! That's awesome. Thanks for confirming! > I am not that much experienced, but that's what I did: > > #) svn update -r 335059 > #) applied your patch > #) recompiled world and kernel > #) reinstalled world and kernel > #) reinstalled my jail's basejail (ezjail technology) > #) reboot > -> return of syslogd clients being able to send syslog messages from jails to > host's syslogd > > I hope that this is sufficient prove that your patch will work? Yes, that's all right. I've just committed the patch (r335314) and will MFC it to 11-STABLE before the end of the week. Best regards, -- Ed Schouten Nuxi, 's-Hertogenbosch, the Netherlands ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
