See: http://scientificlinuxforum.org/index.php?showtopic=1634
Low latency Intel SSD or STEC Zeus or an Intel SSDPEDPX800G3 used as a ZIL either by themselves or in a striped pair when combined with NFS as the use case all yielded the same performance. The key here is that whatever you use it needs to be a device with a very low latency capable of a large amount of IOPS. You will note that when I used SSD in a similar manner of BTRFS the performance was either equal or greater in some areas. As to the Sun storage appliances note the hardware I used ;-) - Chris From: Paul Robert Marino [mailto:[email protected]] Sent: Wednesday, July 10, 2013 5:53 PM To: Brown, Chris (GE Healthcare); [email protected] Subject: Re: KVM + NFS Performance (ZFS/BTRFS/EXT4) Interesting however I have to ask why didn't you use a flash backed ram drive for the ZIL . There is a well know core fundamental design conflict between ZFS and NFS that's so sever that Suns purposely built ZFS based NAS appliances include one of these very expensive (because all of the ones on the market use DDR2) ram drives on a PCI card just to handle the ZIL. The issue is that there is a portion of how NFS handles locking which causes the ZIL to do a shocking number of IOPS on the ZIL (sometimes in the multiple hundreds) per NFS IOP. Even the forks like Free NAS have a list of these cards and sata drive on their site. Note this is only the case with combining ZFS and NFS. -- Sent from my HP Pre3 ________________________________ On Jul 3, 2013 2:23 PM, Brown, Chris (GE Healthcare) <[email protected]<mailto:[email protected]>> wrote: I have done a fair bit of testing and experimenting to maximize KVM VM performance on ZFS. Hopefully someone finds the below useful. Guests/Benchmark used: WinXP Pro + Crystal Disk Mark 1 pass at 1000MG SL 6.4 + Gnome Disk Utility (palimpsest) R/W disk benchmark function Each guest uses a file backed sparse raw disk located on an NFS mounted directory ("/foo"). The NFS mounted directory is interconnected via a dual 10Gbe Twin-ax copper from the targeted storage server disk subsystem and filesystem. The connection is bonded using Linux bonding (mode=4 miimon=250 xmit_hash_policy=layer2+3). NFS server export options were (rw,async,no_root_squash). The NFS client was left with default mount options. The NFS version used was NFSv3 only. NFSv4 was disabled on the storage server via "RPCNFSDARGS = -N 4". *NOTES* ZFS, EXT4, MD+LVM, or BTRFS disk subsystem configuration details are specified in each below test configuration. Low latency Intel SSD models were utilized in all cases wherein SSD are being used. When ext4 is used on top of a zvol with default mount options the journal/metadata is synchronous and therefore offloaded to the ZIL. WinXP qemu-kvm commandline: /usr/libexec/qemu-kvm -name winxpfoo -M rhel6.4.0 -cpu qemu64,-svm -enable-kvm -m 3072 -smp 2,sockets=1,cores=2,threads=1 -nodefconfig -nodefaults -drive file=/foo/winxp.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0 -spice port=5900,addr=3.57.109.210,disable-ticketing -k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 & SL 6.4 qemu-kvm commandline: /usr/libexec/qemu-kvm -name sl64 -M rhel6.4.0 -cpu qemu64,-svm -enable-kvm -m 3072 -smp 2,sockets=1,cores=2,threads=1 -nodefconfig -nodefaults -drive file=/vmstore/foo/SL64hd0.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/vmstore/foo/SL64hd1.img,if=none,id=drive-virtio-disk1,format=raw,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 -spice port=5900,addr=3.57.109.210,disable-ticketing -k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 & -- Test Config #1 -- Storage Server: Sunfire x4140 RAM: 32GB CPU: 2x AMD Opteron 2439SE HBA: LSI 3801 Disks: 8 146GB 10K 2.5" SAS Disk Config: LSI HW IME Raid + Linux LVM Filesystem: LVM+ext4 Network: Intel x520DA2 dual port 10Gbe OS: SL 6.4 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Filesystem: LSI HW Raid+ ext4 Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Benchmark Results (MB/s): **** WinXP **** seq: 295.4 189.7 512k: 298.8 188.9 4K: 14.75 12.99 4K QD32: 16.16 14.42 **** SL 6.4 (MB/s) **** Min Read: 654.3 Max Read: 704.8 Avg Read: 682.5 Min Write: 233.6 Max Write: 551.1 Avg Write: 465.8 AVG Access: 0.5ms -- End Test Config #1 -- -- Test Config #2 -- Storage Server: Sunfire x4270m2 RAM: 72GB CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 24 300GB 10K 2.5" SAS Disks: 4 128GB Intel SATA SSD Disk Config: ZFS Raid10 + Raid 0 ZIL + Raid 0 L2ARC Filesystem: ZFS ZVOL+ext4 Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/ZFS Configuration Details/Commands zpool create das0 mirror <dev> <dev> mirror <dev> <dev>... log <ssd> <ssd> cache <ssd> <ssd> zfs create -s -V 100G -o volblocksize=64K das0/foo mkfs.ext4 -L foo /dev/das0/foo mount /dev/das0/foo /zvol/foo -o noatime **** XP (MB/s) **** seq: 315.4 273.4 512k: 302.4 263.0 4K: 12.94 8.688 4K QD32: 14.45 13.13 **** SL 6.4 (MB/s)**** Min Read: 630.0 Max Read: 698.2 Avg Read: 662.8 Min Write: 357.6 Max Write: 1.1 GB/s Avg Write: 895.3 AVG Access: 0.5ms -- End Test Config #2 -- -- Test Config #3 -- Storage Server: Sunfire x4270m2 RAM: 72GB CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 24 300GB 10K 2.5" SAS Disks: 4 128GB Intel SATA SSD Disk Config: ZFS Raid10 + Raid 0 ZIL + Raid 0 L2ARC Filesystem: ZFS Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/ZFS Configuration Details/Commands zpool create das0 mirror <dev> <dev> mirror <dev> <dev>... log <ssd> <ssd> cache <ssd> <ssd> zfs create das0/foo **** XP (MB/s) **** seq: 111.4 57.36 512k: 95.64 54.33 4K: 11.80 2.878 4K QD32: 11.31 3.095 -- End Test Config #3 -- -- Test Config #4 -- Storage Server: Sunfire x4170 RAM: 72GB CPU:2x Intel E5540 HBA: LSI 92118i Disks: 4 200GB Intel SATA SSD Disks: 2 100 GB Intel SATA SSD Disk Config: ZFS Raid10 + ZIL + L2ARC Filesystem: ZFS ZVOL+ext4 Network: Intel x520DA2 dual port 10Gbe OS: Fedora 18 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/ZFS Configuration Details/Commands zpool create das0 mirror <dev> <dev> mirror <dev> <dev> log <ssd> cache <ssd> zfs create -s -V 100G -o volblocksize=64K das0/foo mkfs.ext4 -L foo /dev/das0/foo mount /dev/das0/foo /zvol/foo -o noatime **** XP (MB/s) **** seq: 196.5 200.2 512k: 195.2 188.8 4K: 9.665 7.043 4K QD32: 12.02 8.140 **** SL 6.4 (MB/s) **** Min Read: 423.7 Max Read: 575.3 Avg Read: 526.7 Min Write: 44.4 Max Write: 668.8 Avg Write: 540.5 AVG Access: 1.4ms -- End Test Config #4 -- -- Test Config #5 -- Storage Server: Sunfire x4170 RAM: 72GB CPU:2x Intel E5540 HBA: LSI 92118i Disks: 4 200GB Intel SATA SSD Disks: 2 100 GB Intel SATA SSD Disk Config: ZFS Raid10 + ZIL + L2ARC Filesystem: ZFS Network: Intel x520DA2 dual port 10Gbe OS: Fedora 18 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/ZFS Configuration Details/Commands zpool create das0 mirror <dev> <dev> mirror <dev> <dev> log <ssd> cache <ssd> zfs create das0/foo **** XP (MB/s) **** seq: 152.5 126.2 512k: 147.4 121.8 4K: 9.189 7.481 4K QD32: 11.90 8.003 **** SL 6.4 (MB/s) **** Min Read: 409.3 Max Read: 564.4 Avg Read: 511.6 Min Write: 237.1 Max Write: 658.0 Avg Write: 555.3 AVG Access: 2.9ms -- End Test Config #5 -- -- Test Config #6 -- Storage Server: Sunfire x4170 RAM: 72GB CPU:2x Intel E5540 HBA: LSI 92118i Disks: 4 200GB Intel SATA SSD Disks: 2 100 GB Intel SATA SSD Disk Config: Linux MD Raid10 + LVM + ext4 Filesystem: ext4 Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/ext4 Configuration Details/Commands 4 Disk SSD Linux MD Raid 10 Stripe Size 1MB 100GB LV(foo) from VG0 via PV MD device (ext4 data ) 2 Disk SSD Linux MD Raid 0 Stripe Size 1MB 1GB LV(journaldev) from VG1 via PV MD device (ext4 journal/metadata) ext4 filesystem lv-foo, external journal on lv-journaldev (external journal_data_ordered) mounted at /vg0/foo (mount options rw,noatime,journal_checksum,journal_async_commit) **** XP (MB/s) **** seq: 251.3 203.9 512k: 249.0 217.7 4K: 10.12 9.012 4K QD32: 13.18 11.54 **** SL 6.4 (MB/s) **** Min Read: 523.3 Max Read: 663.0 Avg Read: 606.4 Min Write: 239.9 Max Write: 656.0 Avg Write: 563.8 AVG Access: 0.8ms -- End Test Config #6 -- -- Test Config #7 -- Storage Server: Sunfire x4170 RAM: 72GB CPU:2x Intel E5540 HBA: LSI 92118i Disks: 4 200GB Intel SATA SSD Disks: 2 100 GB Intel SATA SSD Disk Config: BTRFS Raid10 data + raid0 metadata Filesystem: BTRFS Network: Intel x520DA2 dual port 10Gbe OS: Fedora 18 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/BTRFS Configuration Details/Commands mkfs.btrfs -L foo -m raid0 <ssd> <ssd> -d raid10 <ssd> <ssd> <ssd> <ssd> btrfs subvolume create /das0/foo **** XP (MB/s) **** seq: 195.4 180.0 512k: 180.8 174.5 4K: 9.898 7.183 4K QD32: 11.87 8.248 **** SL 6.4 (MB/s) **** Min Read: 465.4 Max Read: 554.6 Avg Read: 514.1 Min Write: 450.5 Max Write: 670.3 Avg Write: 525.3 AVG Access: 0.8ms -- End Test Config #7-- -- Test Config #8 -- Storage Server: Sunfire x4170 RAM: 72GB CPU:2x Intel E5540 HBA: LSI 92118i Disks: 4 200GB Intel SATA SSD Disks: 2 100 GB Intel SATA SSD Disk Config: BTRFS Raid10 data + raid0 metadata Filesystem: BTRFS Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/BTRFS Configuration Details/Commands mkfs.btrfs -L foo -m raid0 <ssd> <ssd> -d raid10 <ssd> <ssd> <ssd> <ssd> btrfs subvolume create /das0/foo **** XP (MB/s) **** seq: 232.1 123.4 512k: 243.0 213.6 4K: 9.393 9.170 4K QD32: 12.04 10.74 **** SL 6.4 (MB/s) **** Min Read: 508.0 Max Read: 652.4 Avg Read: 594.4 Min Write: 409.6 Max Write: 670.4 Avg Write: 525.8 AVG Access: 0.5 -- End Test Config #8 -- - DHC
