Further testing revealed
that it wasn't an iSCSI performance issue but a zvol
issue. Testing on a SATA disk locally, I get these
numbers (sequentual write):
UFS: 38MB/s
ZFS: 38MB/s
Zvol UFS: 6MB/s
Zvol Raw: ~6MB/s
ZFS is nice and fast but Zvol performance just drops
off a cliff. Suggestion or observations by others
using zvol would be extremely helpful.
# zfs create -V 1g data/zvol-test
# time dd if=/data/media/sol-10-u2-ga-x86-dvd.iso
of=/dev/zvol/rdsk/data/zvol-test bs=32k count=10000
10000+0 records in
10000+0 records out
0.08u 9.37s 2:21.56 6.6%
That's ~ 2.3 MB/s.
I do see *frequent* DKIOCFLUSHWRITECACHE ioctls
(one flush write cache ioctl after writing ~36KB of data, needs ~6-7
milliseconds per flush):
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e02778, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 5736778
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e027c0, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6209599
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e02808, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6572132
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e02850, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6732316
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e02898, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6175876
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e028e0, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6251611
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e02928, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 7756397
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e02970, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6393356
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e029b8, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6147003
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e02a00, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6247036
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e02a48, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6061991
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e02a90, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6284297
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e02ad8, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6174818
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5e02b20, count 9000
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6245923
nsec, error 0
dtrace with stack backtraces:
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5d1ec10, count 9000
0 39404 zio_ioctl:entry
zfs`zil_flush_vdevs+0x144
zfs`zil_commit+0x311
zfs`zvol_strategy+0x4bc
genunix`default_physio+0x308
genunix`physio+0x1d
zfs`zvol_write+0x22
genunix`cdev_write+0x25
specfs`spec_write+0x4d6
genunix`fop_write+0x2e
genunix`write+0x2ae
unix`sys_sysenter+0x104
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6638189
nsec, error 0
0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn
5d1ec58, count 9000
0 39404 zio_ioctl:entry
zfs`zil_flush_vdevs+0x144
zfs`zil_commit+0x311
zfs`zvol_strategy+0x4bc
genunix`default_physio+0x308
genunix`physio+0x1d
zfs`zvol_write+0x22
genunix`cdev_write+0x25
specfs`spec_write+0x4d6
genunix`fop_write+0x2e
genunix`write+0x2ae
unix`sys_sysenter+0x104
0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 7881400
nsec, error 0
=====================================================================
#!/usr/sbin/dtrace -s
BEGIN
{
DKIOC = 0x04 << 8;
DKIOCFLUSHWRITECACHE = DKIOC|34;
}
fbt::bdev_strategy:entry
{
bp = (struct buf *)arg0;
printf("edev %x, flags %x, bn %x, count %x",
bp->b_edev, bp->b_flags, bp->_b_blkno._p._l, bp->b_bcount);
}
fbt::zio_ioctl:entry
/arg3 == DKIOCFLUSHWRITECACHE/
{
flushtime = timestamp;
}
fbt::vdev_disk_ioctl_done:entry
/flushtime/
{
this->zio = (zio_t *)arg0;
printf("%s time: %llu nsec, error %d",
this->zio->io_cmd == DKIOCFLUSHWRITECACHE ? "DKIOCFLUSHWRITECACHE" :
"???",
timestamp - flushtime, arg1);
flushtime = 0;
}
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
