On Wed, 2006-05-03 at 22:16, Joe Little wrote:
> Are there any scripts to track NFS server performance? More
> specifically, I'm seeing terrible performance from Linux 2.6 clients
> against Nevada (B38) with a ZFS shared pool, but only with directorys
> with many small files. The performance for large files seems
> reasonable (20MB/sec) but small drops down to low k's/sec on copying
> over NFS.
Perhaps not your problem but yesterday, Dave Miner and I traced some
astoundingly poor NFS-over-ZFS performance on the Netra X1 to an
apparent bug in sparc ATA driver code.
The interactive response on the server while a tar extract (of the
firefox source tarball, so lots of small files) was running on a
(solaris) client reminded me of what it was like to use an overloaded
timeshared TOPS-20 system the night before problem sets were due...
tar was writing about one file per second.
mpstat or vmstat (I forget which one I used) showed 100% system time.
It took a few tries (one failed because dtrace aborted due to systemic
unresponsiveness; the second started right before tar finished and gave
us a nice profile of the idle loop..), but eventually we got useful data
from:
# lockstat -s 10 -I sleep 10
The system was so overloaded that "sleep 10" took 22.7 seconds
essentially all cpu time was spent in stack traces resembling:
Count indv cuml rcnt nsec CPU+PIL Caller
1004 42% 42% 0.00 3322 cpu[0] drv_usecwait+0x78
nsec ------ Time Distribution ------ count Stack
4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 975 ata_wait+0x90
8192 | 22
acersb_get_intr_status+0x34
16384 | 7 ata_set_feature+0x124
ata_disk_start+0x15c
ata_hba_start+0xbc
The driver is busy-waiting in drv_usecwait() for long periods of time,
and my suspicion is that this is most likely an artifact of ZFS's
regular disk write cache flushes.
we filed:
6421427 netra x1 slagged by NFS over ZFS leading to long spins in the
ATA driver code
(if you see unreasonably slow performance, essentially no idle time, and
high system time when you think you shouldn't, the above use of
lockstat's profiling mechanism or dtrace equivalents thereof aggregating
on stack() will point out where in the code the kernel is spinning its
wheels.)
- Bill
