-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/150/
-----------------------------------------------------------
Review request for OpenZFS Developer Mailing List and Christopher Siden.
Summary (updated)
-----------------
5498 kmem_reap does one xcall per page
Bugs: 5498
https://www.illumos.org/projects/illumos-gate//issues/5498
Repository: illumos-gate
Description (updated)
-------
Author: Matthew Ahrens <[email protected]>
Date: Tue Dec 30 11:01:02 2014 -0800
5498 kmem_reap does one xcall per page
Reviewed by: Adam Leventhal <[email protected]>
Reviewed by: George Wilson <[email protected]>
Reviewed by: Paul Dagnelie <[email protected]>
Reviewed by: Dan Kimmel <[email protected]>
We see performance problems due to kmem_reap() of ZFS's zio buf caches. The
kmem_reap() causes a lot of cross-calls to tear down TLB entries, which is
slow. It can also cause latency bubbles in concurrent zfs_read/zfs_write
operations.
The problem is reliably triggered when a workload switches between accessing
files of different blocksizes. This causes kmem to need to free lots of buffers
of the old size in order to allocate ones of the new size.
We can improve on this in some cases by using one xcall per slab, to tear down
all the virtually contiguous pages used to back the slab. The performance
benefit will be most noticeable when reaping larger (e.g. 128KB) caches.
Diffs (updated)
-----
usr/src/uts/i86pc/vm/htable.c 665eb9d41407924de0db2f1781213b7f7a3364e3
usr/src/uts/i86pc/vm/htable.h c0f5639c60c8c5e4ff9a3c8f8960fb6e877c7bd7
usr/src/uts/i86pc/vm/hat_i86.c 64e1c2fb46b278bf45070d8fabb06486b7347751
usr/src/uts/i86pc/vm/hat_i86.h 03317b35e849720f5c0115cc3dc86c5613759f71
Diff: https://reviews.csiden.org/r/150/diff/
Testing (updated)
-------
ztest
zfs test suite
internal link: http://jenkins/job/zfs-precommit/1466/console
performance test with the following script:
```
#!/bin/bash -x
if [[ ! -f /test/big/file ]]; then
zfs create -o recordsize=128k test/big
zfs create -o recordsize=8k test/small
dd if=/dev/zero of=/test/big/file bs=1024k count=200000
dd if=/dev/zero of=/test/small/file bs=1024k count=200000
fi
/home/mahrens/reap.d &
dd if=/test/big/file of=/dev/null bs=1024k
dd if=/test/big/file of=/dev/null bs=1024k
dd if=/test/small/file of=/dev/null bs=1024k
dd if=/test/small/file of=/dev/null bs=1024k
```
results: 4X performance improvement in kmem_reap of 128K cache.
Have not tested with __xpv (fully paravirtualized Xen). Can anyone help me
with this testing? I don't have a setup for this. Build artifacts are
available at:
http://mahrens.org/xcall/packages.tgz
http://mahrens.org/xcall/proto.tgz
Thanks,
Matthew Ahrens
_______________________________________________
developer mailing list
[email protected]
http://lists.open-zfs.org/mailman/listinfo/developer
