On 5/24/20 01:29, info+varn...@shee.org wrote: > This notes > > https://varnish-cache.org/docs/trunk/installation/platformnotes.html > > has a comment about "Transparent hugepages". > > Does this still apply to EL8?
That's a good heads-up that those docs need to be updated -- they refer to RHEL6 and Linux kernel 3.2. If I'm not mistaken, enabling THP by default was fairly new at the time, but it's still the default and that's old news now, as your settings confirmed (just checked that it's also the default on my Debian stretch laptop). The issue is not really the distro or kernel version, but the use of the THP feature, and it's still a problem, probably always will be. AFAICT THP does nothing good for Varnish. It's harmless if you're lucky, but it can be very disruptive. I haven't tried it with RHEL8. The doc says that it "is known to cause sporadic crashes of Varnish", but while I haven't seen crashes, on RHEL7 I've seen that the memory usage of the cache process bloats up enormously, orders of magnitude larger than the actual size of the cache and anything else in Varnish that occupies memory. After disabling THP for Varnish (as detailed below), I saw memory usage become much smaller, more like what you'd expect from the cache size and other overhead. There's an explanation for why THP causes that, but suffice it to say that THP creates trouble for a variety of apps that manage a lot of memory. MongoDB, Oracle, redis and many other projects advise you to turn it off. THP is inevitably a problem for the jemalloc memory allocator, which is invariably used with Varnish. You can turn off THP system-wide with: $ echo never > /sys/kernel/mm/transparent_hugepage/enabled Or, I believe this may work in /etc/grub.conf: transparent_hugepage=never Since you're on RHEL8, you also have the option of disabling THP for jemalloc when used by Varnish; so you don't have to turn it off for everything, if you prefer to leave on the default for other processes on your system. The option is: thp:never One way to do that is to start Varnish with that setting in MALLOC_CONF in its environment: $ MALLOC_CONF=thp:never /usr/sbin/varnishd -a :80 ... Or you could set thp:never in /etc/malloc.conf, in which case the setting holds for any app that uses jemalloc. The jemalloc man page has all the details. This is possible in RHEL8 because el8 supports versions of jemalloc that have the option. Earlier versions of jemalloc didn't have it, in particular 3.0.6, on which the world was stuck for a very long time, and was the latest available in el7. For readers who are using el7/RHEL7 -- I patched up an RPM that installs more recent jemalloc on el7. Git repo here: https://code.uplex.de/uplex-varnish/libjemalloc2-el7-rpm (Still no README there, my bad.) There's an el7 package repo with the RPM at https://pkg.uplex.de/: $ yum-config-manager --add-repo https://pkg.uplex.de/rpm/7/uplex-varnish/x86_64/ Newer versions of jemalloc have a different SO name, libjemalloc.so.2, whereas most software built for el7 that links to jemalloc expects libjemalloc.so.1. That includes the el7 Varnish RPMs from packagecloud. So if you have that in your Varnish binary (if ldd points to libjemalloc.so.1), and you want to use the el7 RPM for newer jemalloc, you can do this: $ patchelf --replace-needed libjemalloc.so.1 libjemalloc.so.2 /path/to/varnishd Then check ldd, you should see it now pointing to libjemalloc.so.2. I have that working in production, which made the thp:never setting possible, and that got rid of the memory bloat. Sorry for the long-winded response, I had a big fight with this problem a few months ago, and am still a little miffed at transparent hugepages. HTH, Geoff PS while I'm on the subject, shout-out to Ingvar Hagelund, who does the epel/fedora packaging for Varnish, jemalloc, and a variety of other things. He did the hard work packaging jemalloc, I just changed a version number. -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstraße 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de
signature.asc
Description: OpenPGP digital signature
_______________________________________________ varnish-misc mailing list varnish-misc@varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc