> > As a random guess, try pointing PHP tmp directory to
> > /var/tmp (backed by zfs) and see if any behaviors change?
> > Good luck,
> > //Jim
> Thanks for your suggestions. Actually the default PHP tmp directory
> was /var/tmp, and I changed "/var/tmp" to "/tmp". This reduced zfs
> root lock contention significantly. However, I still see a bunch
> of lock
> contention. So I'm here ask for help.
Well, as a further attempt down this road, is it possible for you to rule out
ZFS from swapping - i.e. if RAM amounts permit, disable the swap at all
(swap -d /dev/zvol/dsk/rpool/swap) or relocate it to dedicated slices of
same or better yet separate disks?
If you do have lots of swapping activity (that can be seen in "vmstat 1" as
si/so columns) going on in a zvol, you're likely to get much fragmentation
in the pool, and searching for contiguous stretches of space can become
tricky (and time-consuming), or larger writes can get broken down into
many smaller random writes and/or "gang blocks", which is also slower.
At least such waiting on disks can explain the overall large kernel times.
You can also see the disk wait times ratio in "iostat -xzn 1" column "%w"
and disk busy times ratio in "%b" (second and third from the right).
I dont't remember you posting that.
If these are accounting in tens, or even close or equal to 100%, then
your disks are the actual bottleneck. Speeding up that subsystem,
including addition of cache (ARC RAM, L2ARC SSD, maybe ZIL
SSD/DDRDrive) and combatting fragmentation by moving swap and
other scratch spaces to dedicated pools or raw slices might help.
zfs-discuss mailing list