Just wanted to check in to see if anybody had any recommendations for tuning parameters for 10GbE performance or iSCSI.
I have 2 Myricom 10GbE adapters direct connected to one another. On the SmartOS side, I have an iSCSI ZFS volume set up. On the Mac side, I’m using GlobalSAN to attach to the SmartOS volume. Jumbo frames (MTU=9000) are enabled on both sides. I’ve benchmarked my performance using netperf, and I’m seeing the following (executed from the Mac side, netperf server running on SmartOS): # netperf -H 192.168.2.2 -t TCP_STREAM -C -c -l 60 -- -s 512K -S 512K MIGRATED TCP STREAM TEST from (null) (0.0.0.0) port 0 AF_INET to (null) () port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % O % ? us/KB us/KB 524744 524288 524288 60.00 5512.56 6.35 10.42 1.132 -0.310 I’ve also created a very large file of 0’s, and am using “dd” to copy them over. Here’s what I’m seeing when running that on the Mac: # time dd if=junk.zero of=/Volumes/remote/junk.zero bs=1048576 43158+1 records in 43158+1 records out 45254967296 bytes transferred in 92.342684 secs (490076369 bytes/sec) real 1m32.382s user 0m0.041s sys 0m30.624s I’ve followed a few tuning guides on the Mac, which actually brought the numbers up to the levels I’m showing here. I’m now looking for things on the SmartOS side that can help. BTW, I did try the suggestions here (https://community.emc.com/docs/DOC-39156 <https://community.emc.com/docs/DOC-39156>), but changing those properties didn’t seem to make a difference to any of my numbers. According to my Mac, I have good throughput to the Myricom card itself (a value of 1280 corresponds to 10Gb/sec), so I don’t think that there’s an issue between the Mac and my ethernet card. The card is in a Mercury Helios external cage, connected via Thunderbolt 2 (20Gbs top speed). The card is a Myricom 10G-PCIE2-8B2-2S NIC. # sysctl net.myri10ge | grep dma net.myri10ge.en13.dma_read_bw_MBs: 1436 net.myri10ge.en13.dma_write_bw_MBs: 1456 net.myri10ge.en13.dma_read_write_bw_MBs: 2610 net.myri10ge.en12.dma_read_bw_MBs: 1436 net.myri10ge.en12.dma_write_bw_MBs: 1456 net.myri10ge.en12.dma_read_write_bw_MBs: 2610 Finally, the SmartOS system itself is a SuperMicro X8DTE-F running 2 Xeon L5630’s (16 cores total) with 96GB of ECC memory and a three 3TB hard drives, in a 3-way mirror, that the iSCSI volume is on. Synchronization is disabled: [root@smartos ~]# zfs get sync zones/zpool/iscsi-1 NAME PROPERTY VALUE SOURCE zones/zpool/iscsi-1 sync disabled local Sorry for the long post, but trying to supply any pertinent information without people having to ask for it. Any help in boosting these numbers would be appreciated. Thanks, John ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
