On 5/4/06, Roch Bourbonnais - Performance Engineering <Roch.Bourbonnais at sun.com> wrote: > > Joe Little writes: > > In my case, I'm using an iscsi backend. However, doing any operations > > locally are fast via ZFS, even with small files. If I setup an rsyncd > > service, clients can also write small files relatively fast > > (3-5MB/sec), but over NFS its 5-64K/sec. So, there is a major problem > > still with NFS/ZFS interaction. > > > > Does rsync follow an open/write/close pattern ? > Because of NFS close-to-open consistency, that pattern > delivers at best a throughput of > > (avg file size) / (server IO latency) > > Is your avg file size is < 640B and latency 0.01 sec ? > This is true no matter what the backend FS. Or have you > observed something different with say UFS ?
I'll try the nocto option. Our average file size (I believe) is in the low K's -- perhaps between 1K-8K. An example where this gets really slow is a prof with untarred versions of every linux kernel release for source code analysis. It processes these directories about 2K/sec. I will confess to not knowing how to measure latency. I haven't observed the issue with UFS, but I haven't been trying much since UFS doesn't interest me as much. That said, our UFS only Solaris 9/10 machines haven't had this issue. > > Mounting with the -nocto option (at your own risk) may help > if this is the issue. > > > -r > > > > > On 5/4/06, Bill Sommerfeld <sommerfeld at sun.com> wrote: > > > On Wed, 2006-05-03 at 22:16, Joe Little wrote: > > > > Are there any scripts to track NFS server performance? More > > > > specifically, I'm seeing terrible performance from Linux 2.6 clients > > > > against Nevada (B38) with a ZFS shared pool, but only with directorys > > > > with many small files. The performance for large files seems > > > > reasonable (20MB/sec) but small drops down to low k's/sec on copying > > > > over NFS. > > > > > > Perhaps not your problem but yesterday, Dave Miner and I traced some > > > astoundingly poor NFS-over-ZFS performance on the Netra X1 to an > > > apparent bug in sparc ATA driver code. > > > > > > The interactive response on the server while a tar extract (of the > > > firefox source tarball, so lots of small files) was running on a > > > (solaris) client reminded me of what it was like to use an overloaded > > > timeshared TOPS-20 system the night before problem sets were due... > > > > > > tar was writing about one file per second. > > > > > > mpstat or vmstat (I forget which one I used) showed 100% system time. > > > > > > It took a few tries (one failed because dtrace aborted due to systemic > > > unresponsiveness; the second started right before tar finished and gave > > > us a nice profile of the idle loop..), but eventually we got useful data > > > from: > > > > > > # lockstat -s 10 -I sleep 10 > > > > > > The system was so overloaded that "sleep 10" took 22.7 seconds > > > > > > essentially all cpu time was spent in stack traces resembling: > > > > > > Count indv cuml rcnt nsec CPU+PIL Caller > > > 1004 42% 42% 0.00 3322 cpu[0] drv_usecwait+0x78 > > > > > > nsec ------ Time Distribution ------ count Stack > > > 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 975 ata_wait+0x90 > > > 8192 | 22 > acersb_get_intr_status+0x34 > > > 16384 | 7 > ata_set_feature+0x124 > > > > ata_disk_start+0x15c > > > ata_hba_start+0xbc > > > > > > The driver is busy-waiting in drv_usecwait() for long periods of time, > > > and my suspicion is that this is most likely an artifact of ZFS's > > > regular disk write cache flushes. > > > > > > we filed: > > > > > > 6421427 netra x1 slagged by NFS over ZFS leading to long spins in the > > > ATA driver code > > > > > > (if you see unreasonably slow performance, essentially no idle time, and > > > high system time when you think you shouldn't, the above use of > > > lockstat's profiling mechanism or dtrace equivalents thereof aggregating > > > on stack() will point out where in the code the kernel is spinning its > > > wheels.) > > > > > > - Bill > > > > > > > > _______________________________________________ > > dtrace-discuss mailing list > > dtrace-discuss at opensolaris.org > >
