On Wed, Dec 05, 2018 at 11:49:26AM -0800, David Rientjes wrote: > High thp utilization is not always better, especially when those hugepages > are accessed remotely and introduce the regressions that I've reported. > Seeking high thp utilization at all costs is not the goal if it causes > workloads to regress.
Is it possible what you need is a defrag=compactonly_thisnode to set instead of the default defrag=madvise? The fact you seem concerned about page fault latencies doesn't make your workload an obvious candidate for MADV_HUGEPAGE to begin with. At least unless you decide to smooth the MADV_HUGEPAGE behavior with an mbind that will simply add __GFP_THISNODE to the allocations, perhaps you'll be even faster if you invoke reclaim in the local node for 4k allocations too. It looks like for your workload THP is a nice to have add-on, which is practically true of all workloads (with a few corner cases that must use MADV_NOHUGEPAGE), and it's what the defrag= default is about. Is it possible that you just don't want to shut off completely compaction in the page fault and if you're ok to do it for your library, you may be ok with that for all other apps too? That's a different stance from other MADV_HUGEPAGE users because you don't seem to mind a severely crippled THP utilization in your app. With your patch the utilization will go down a lot compared to the previous __GFP_THISNODE swap storm capable and you're still very fine with that. The fact you're fine with that points in the direction of changing the default tuning for defrag= to something stronger than madvise (that is precisely the default setting that is forcing you to use MADV_HUGEPAGE to get a chance to get some THP once a in a while during the page fault, after some uptime). Considering mbind surprisingly isn't privileged (so I suppose it may cause swap storms equivalent to __GFP_THISNODE if maliciously used after all) you could even consider a defrag=thisnode to force compaction+defrag local to the node to retain your THP+NUMA dynamic partitioning behavior that ends up swappin heavy in the local node.