On 18.11.2013 10:41, Adrian Chadd wrote:
Your patch does three things:

* adds a couple new buckets;

These new buckets make bucket size self-tuning more soft and precise. Without them there are buckets for 1, 5, 13, 29, ... items. While at bigger sizes difference about 2x is fine, at smallest ones it is 5x and 2.6x respectively. New buckets make that line look like 1, 3, 5, 9, 13, 29, reducing jumps between steps, making algorithm work softer, allocating and freeing memory in better fitting chunks. Otherwise there is quite a big gap between allocating 128K and 5x128K of RAM at once.

* reduces some lock contention

More precisely patch adds check for congestion on free to grow bucket sizes same as on allocation. As consequence that indeed should reduce lock congestion, but I don't have specific numbers. All I see is that VM and UMA mutexes no longer appear in profiling top after all these changes.

* does soft back pressure

In this list you have missed mentioning small but major point of the patch -- we should prevent problems, not just solve them. As I have written in original email, this specific change shown me 1.5x performance improvement in low-memory condition. As I understand, that happened because VM no longer have to repeatedly allocate and free hugely oversized buckets of 10-15 * 128K.

* does the aggressive backpressure.

After all above that is mostly just a safety belt. With 40GB RAM that code was triggered only couple times during full hour of testing with debug logging inserted there. On machine with 2GB RAM it is triggered quite regularly and probably that is unavoidable since even with lowest bucket size of one item 24 CPUs mean 48 cache buckets, i.e. up to 6MB of otherwise unreleasable memory for single 128K zone.

So, do you get any benefits from just the first one, or first two?

I don't see much reason to handle that in pieces. As I have described above, each part has own goal, but they much better work together.

On 17 November 2013 15:09, Alexander Motin <m...@freebsd.org> wrote:

I've created patch, based on earlier work of avg@, to add back pressure to
UMA allocation caches. The problem of physical memory or KVA exhaustion
existed there for many years and it is quite critical now for improving
systems performance while keeping stability. Changes done in memory
allocation last years improved situation. but haven't fixed completely. My
patch solves remaining problems from two sides: a) reducing bucket sizes
every time system detects low memory condition; and b) as last-resort
mechanism for very low memory condition, it cycling over all CPUs to purge
their per-CPU UMA caches. Benefit of this approach is in absence of any
additional hard-coded limits on cache sizes -- they are self-tuned, based on
load and memory pressure.

With this change I believe it should be safe enough to enable UMA allocation
caches in ZFS via vfs.zfs.zio.use_uma tunable (at least for amd64). I did
many tests on machine with 24 logical cores (and as result strong allocation
cache effects), and can say that with 40GB RAM using UMA caches, allowed by
this change, by two times increases results of SPEC NFS benchmark on ZFS
pool of several SSDs. To test system stability I've run the same test with
physical memory limited to just 2GB and system successfully survived that,
and even showed results 1.5 times better then with just last resort measures
of b). In both cases tools/umastat no longer shows unbound UMA cache growth,
that makes me believe in viability of this approach for longer runs.

I would like to hear some comments about that:

Alexander Motin
freebsd-current@freebsd.org mailing list
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Reply via email to