Joe Little writes:
> In my case, I'm using an iscsi backend. However, doing any operations
> locally are fast via ZFS, even with small files. If I setup an rsyncd
> service, clients can also write small files relatively fast
> (3-5MB/sec), but over NFS its 5-64K/sec. So, there is a major problem
> still with NFS/ZFS interaction.
>
Does rsync follow an open/write/close pattern ?
Because of NFS close-to-open consistency, that pattern
delivers at best a throughput of
(avg file size) / (server IO latency)
Is your avg file size is < 640B and latency 0.01 sec ?
This is true no matter what the backend FS. Or have you
observed something different with say UFS ?
Mounting with the -nocto option (at your own risk) may help
if this is the issue.
-r
>
> On 5/4/06, Bill Sommerfeld <sommerfeld at sun.com> wrote:
> > On Wed, 2006-05-03 at 22:16, Joe Little wrote:
> > > Are there any scripts to track NFS server performance? More
> > > specifically, I'm seeing terrible performance from Linux 2.6 clients
> > > against Nevada (B38) with a ZFS shared pool, but only with directorys
> > > with many small files. The performance for large files seems
> > > reasonable (20MB/sec) but small drops down to low k's/sec on copying
> > > over NFS.
> >
> > Perhaps not your problem but yesterday, Dave Miner and I traced some
> > astoundingly poor NFS-over-ZFS performance on the Netra X1 to an
> > apparent bug in sparc ATA driver code.
> >
> > The interactive response on the server while a tar extract (of the
> > firefox source tarball, so lots of small files) was running on a
> > (solaris) client reminded me of what it was like to use an overloaded
> > timeshared TOPS-20 system the night before problem sets were due...
> >
> > tar was writing about one file per second.
> >
> > mpstat or vmstat (I forget which one I used) showed 100% system time.
> >
> > It took a few tries (one failed because dtrace aborted due to systemic
> > unresponsiveness; the second started right before tar finished and gave
> > us a nice profile of the idle loop..), but eventually we got useful data
> > from:
> >
> > # lockstat -s 10 -I sleep 10
> >
> > The system was so overloaded that "sleep 10" took 22.7 seconds
> >
> > essentially all cpu time was spent in stack traces resembling:
> >
> > Count indv cuml rcnt nsec CPU+PIL Caller
> > 1004 42% 42% 0.00 3322 cpu[0] drv_usecwait+0x78
> >
> > nsec ------ Time Distribution ------ count Stack
> > 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 975 ata_wait+0x90
> > 8192 | 22
> > acersb_get_intr_status+0x34
> > 16384 | 7 ata_set_feature+0x124
> > ata_disk_start+0x15c
> > ata_hba_start+0xbc
> >
> > The driver is busy-waiting in drv_usecwait() for long periods of time,
> > and my suspicion is that this is most likely an artifact of ZFS's
> > regular disk write cache flushes.
> >
> > we filed:
> >
> > 6421427 netra x1 slagged by NFS over ZFS leading to long spins in the
> > ATA driver code
> >
> > (if you see unreasonably slow performance, essentially no idle time, and
> > high system time when you think you shouldn't, the above use of
> > lockstat's profiling mechanism or dtrace equivalents thereof aggregating
> > on stack() will point out where in the code the kernel is spinning its
> > wheels.)
> >
> > - Bill
> >
> >
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
