This is very well written and quite detailed. It has all the makings of a great post I'd point people to. However, as currently stated, I'd worry that it would (mis)lead readers into using THP with "always" /sys/kernel/mm/transparent_hugepage/defrag settings (instead of "defer"), and/or on older (pre-4.6) kernels with a false sense that the many-msec slow path allocation latency problems many people warn about don't actually exist. You do link to the discussions on the subject, but the measurements and summary conclusion of the posting alone would not end up warning people who don't actually follow those links.
I assume your intention is not to have the reader conclude that "there is lots of advise out there telling you to turn off THP, and it is wrong. Turning it on is perfectly safe, and may significantly speed up your application", but are instead are aiming for something like "THP used to be problematic enough to cause wide ranging recommendations to simply turn it off, but this has changed with recent Linux kernels. It is now safe to use in widely applicable ways (will th the right settings) and can really help application performance without risking huge stalls". Unfortunately, I think that many readers would understand the current text as the former, not the latter. Here is what I'd change to improve on the current text: 1. Highlight the risk of high slow path allocation latencies with the "always" (and even "madvise") setting in /sys/kernel/mm/transparent_hugepage/defrag, the fact that the "defer" option is intended to address those risks, and this defer option is available with Linux kernel versions 4.6 or later. 2. Create an environment that would actually demonstrate these very high (many msec or worse) latencies in the allocation slow path with defrag set to "always". This is the part that will probably take some extra work, but it will also be a very valuable contribution. The issues are so widely reported (into the 100s of msec or more, and with a wide verity of workloads as your links show) that intentional reproduction *should* be possible. And being able to demonstrate it actually happening will also allow you to demonstrate how newer kernels address it with the defer setting. 3. Show how changing the defrag setting to "defer" removes the high latencies seen by the allocation slow path under the same conditions. For (2) above, I'd look to induce a situation where the allocation slow path can't find a free 2MB page without having to defragment one directly. E.g. - I'd start by significantly slowing down the background defragmentation in khugepaged (e.g set /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs to 3600000). I'd avoid turning it off completely in order to make sure you are still measuring the system in a configuration that believes it does background defragmentation. - I'd add some static physical memory pressure (e.g. allocate and touch a bunch of anonymous memory in a process that would just sit on it) such that the system would only have 2-3GB free for buffers and your netty workload's heap. A sleeping jvm launched with an empirically sized and big enough -Xmx and -Xms and with AlwaysPretouch on is an easy way to do that. - I'd then create an intentional and spiky fragmentation load (e.g. perform spikes of a scanning through a 20GB file every minute or so). - with all that in place, I'd then repeatedly launch and run your Netty workload without the PreTouch flag, in order to try to induce situations where an on-demand allocated 2MB heap page hits the slow path, and the effect shows up in your netty latency measurements. All the above are obviously experimentation starting points, and may take some iteration to actually induce the demonstrated high latencies we are looking for. But once you are able to demonstrate the impact of on-demand allocation doing direct (synchronous) compaction both in your application latency measurement and in your kernel tracing data, you would then be able to try the same experiment with the defrag setting set to "defer" to show how newer kernels and this new setting now make it safe (or at least much more safe) to use THP. And with that actually demonstrated, everything about THP recommendations for freeze-averse applications can change, making for a really great posting. Sent from my iPad On Aug 18, 2017, at 3:00 AM, Alexandr Nikitin <[email protected]<mailto:[email protected]>> wrote: I decided to write a post about measuring the performance impact (otherwise it stays in my messy notes forever) Any feedback is appreciated. https://alexandrnikitin.github.io/blog/transparent-hugepages-measuring-the-performance-impact/ On Saturday, August 12, 2017 at 1:01:31 PM UTC+3, Alexandr Nikitin wrote: I played with Transparent Hugepages some time ago and I want to share some numbers based on real world high-load applications. We have a JVM application: high-load tcp server based on netty. No clear bottleneck, CPU, memory and network are equally highly loaded. The amount of work depends on request content. The following numbers are based on normal server load ~40% of maximum number of requests one server can handle. When THP is off: End-to-end application latency in microseconds: "p50" : 718.891, "p95" : 4110.26, "p99" : 7503.938, "p999" : 15564.827, perf stat -e dTLB-load-misses,iTLB-load-misses -p PID -I 1000 ... ... 25,164,369 iTLB-load-misses ... 81,154,170 dTLB-load-misses ... When THP is always on: End-to-end application latency in microseconds: "p50" : 601.196, "p95" : 3260.494, "p99" : 7104.526, "p999" : 11872.642, perf stat -e dTLB-load-misses,iTLB-load-misses -p PID -I 1000 ... ... 21,400,513 dTLB-load-misses ... 4,633,644 iTLB-load-misses ... As you can see THP performance impact is measurable and too significant to ignore. 4.1 ms vs 3.2 ms 99%% and 100M vs 25M TLB misses. I also used SytemTap to measure few kernel functions like collapse_huge_page, clear_huge_page, split_huge_page. There were no significant spikes using THP. AFAIR that was 3.10 kernel which is 4 years old now. I can repeat experiments with the newer kernels if there's interest. (I don't know what was changed there though) On Monday, August 7, 2017 at 6:42:21 PM UTC+3, Peter Veentjer wrote: Hi Everyone, I'm failing to understand the problem with transparent huge pages. I 'understand' how normal pages work. A page is typically 4kb in a virtual address space; each process has its own. I understand how the TLB fits in; a cache providing a mapping of virtual to real addresses to speed up address conversion. I understand that using a large page e.g. 2mb instead of a 4kb page can reduce pressure on the TLB. So till so far it looks like huge large pages makes a lot of sense; of course at the expensive of wasting memory if only a small section of a page is being used. The first part I don't understand is: why is it called transparent huge pages? So what is transparent about it? The second part I'm failing to understand is: why can it cause problems? There are quite a few applications that recommend disabling THP and I recently helped a customer that was helped by disabling it. It seems there is more going on behind the scene's than having an increased page size. Is it caused due to fragmentation? So if a new page is needed and memory is fragmented (due to smaller pages); that small-pages need to be compacted before a new huge page can be allocated? But if this would be the only thing; this shouldn't be a problem once all pages for the application have been touched and all pages are retained. So I'm probably missing something simple. -- You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/sljzehnCNZU/unsubscribe. To unsubscribe from this group and all its topics, send an email to [email protected]<mailto:[email protected]>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
