Am Sonntag, den 24.05.2020, 10:09 +0200 schrieb Geoff Simmons: > On 5/24/20 01:29, [email protected] wrote: > > This notes > > > > https://varnish-cache.org/docs/trunk/installation/platformnotes.html > > > > has a comment about "Transparent hugepages". > > > > Does this still apply to EL8? > > That's a good heads-up that those docs need to be updated -- they > refer > to RHEL6 and Linux kernel 3.2. If I'm not mistaken, enabling THP by > default was fairly new at the time, but it's still the default and > that's old news now, as your settings confirmed (just checked that > it's > also the default on my Debian stretch laptop). > > The issue is not really the distro or kernel version, but the use of > the > THP feature, and it's still a problem, probably always will be. > AFAICT > THP does nothing good for Varnish. It's harmless if you're lucky, but > it > can be very disruptive. > > I haven't tried it with RHEL8. The doc says that it "is known to > cause > sporadic crashes of Varnish", but while I haven't seen crashes, on > RHEL7 > I've seen that the memory usage of the cache process bloats up > enormously, orders of magnitude larger than the actual size of the > cache > and anything else in Varnish that occupies memory. After disabling > THP > for Varnish (as detailed below), I saw memory usage become much > smaller, > more like what you'd expect from the cache size and other overhead. > > There's an explanation for why THP causes that, but suffice it to say > that THP creates trouble for a variety of apps that manage a lot of > memory. MongoDB, Oracle, redis and many other projects advise you to > turn it off. THP is inevitably a problem for the jemalloc memory > allocator, which is invariably used with Varnish. > > You can turn off THP system-wide with: > > $ echo never > /sys/kernel/mm/transparent_hugepage/enabled > > Or, I believe this may work in /etc/grub.conf: > > transparent_hugepage=never > > Since you're on RHEL8, you also have the option of disabling THP for > jemalloc when used by Varnish; so you don't have to turn it off for > everything, if you prefer to leave on the default for other processes > on > your system. The option is: thp:never > > One way to do that is to start Varnish with that setting in > MALLOC_CONF > in its environment: > > $ MALLOC_CONF=thp:never /usr/sbin/varnishd -a :80 ... > > Or you could set thp:never in /etc/malloc.conf, in which case the > setting holds for any app that uses jemalloc. The jemalloc man page > has > all the details. > > This is possible in RHEL8 because el8 supports versions of jemalloc > that > have the option. Earlier versions of jemalloc didn't have it, in > particular 3.0.6, on which the world was stuck for a very long time, > and > was the latest available in el7. > > For readers who are using el7/RHEL7 -- I patched up an RPM that > installs > more recent jemalloc on el7. Git repo here: > > https://code.uplex.de/uplex-varnish/libjemalloc2-el7-rpm > (Still no README there, my bad.) > > There's an el7 package repo with the RPM at https://pkg.uplex.de/: > > $ yum-config-manager --add-repo > https://pkg.uplex.de/rpm/7/uplex-varnish/x86_64/ > > Newer versions of jemalloc have a different SO name, > libjemalloc.so.2, > whereas most software built for el7 that links to jemalloc expects > libjemalloc.so.1. That includes the el7 Varnish RPMs from > packagecloud. > So if you have that in your Varnish binary (if ldd points to > libjemalloc.so.1), and you want to use the el7 RPM for newer > jemalloc, > you can do this: > > $ patchelf --replace-needed libjemalloc.so.1 libjemalloc.so.2 > /path/to/varnishd > > Then check ldd, you should see it now pointing to libjemalloc.so.2. > > I have that working in production, which made the thp:never setting > possible, and that got rid of the memory bloat. > > Sorry for the long-winded response, I had a big fight with this > problem > a few months ago, and am still a little miffed at transparent > hugepages. >
Danke Geoff! for your explanatory answer. It gives some insights that motivates to go further then just disabling a system option. I will take a deeper dive into the current context (EL8) and check what is still relevant and what not and then come back ... -- Leon _______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
