Hello.

I faced with a strange performance problem with new disk shelf.
We a using  ZFS system with SATA disks for a while.
It is Supermicro SC846-E16 chassis, Supermicro X8DTH-6F motherboard with 96Gb RAM and 24 HITACHI HDS723020BLA642 SATA disks attached to onboard LSI 2008 controller.


Pretty much satisfied with it we bought additional shelf with SAS disks for VMs hosting. New shelf is Supermicro SC846-E26 chassis. Disks model is HITACHI HUS156060VLS600 (15K 600Gb SAS2). Additional controller LSI 9205-8e was installed in server and connected with JBOD. I connected JBOD with 2 channels and setup multi path first, but when i noticed performance problem i disabled multi path and disconnected one cable (for sure it is not multipath cause the problem).

Problem description follow:

Creating test pool with 5 pair of mirrors (new shelf, SAS disks)

# zpool create -o version=28 -O primarycache=none test mirror c9t5000CCA02A138899d0 c9t5000CCA02A102181d0 mirror c9t5000CCA02A13500Dd0 c9t5000CCA02A13316Dd0 mirror c9t5000CCA02A005699d0 c9t5000CCA02A004271d0 mirror c9t5000CCA02A004229d0 c9t5000CCA02A1342CDd0 mirror c9t5000CCA02A1251E5d0 c9t5000CCA02A1151DDd0

(primarycache=none) to disable ARC influence


Testing sequential write
# dd if=/dev/zero of=/test/zero bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 1.04272 s, 2.1 GB/s

iostat when writing look like
     r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
0.0 1334.6 0.0 165782.9 0.0 8.4 0.0 6.3 1 86 c9t5000CCA02A1151DDd0 0.0 1345.5 0.0 169575.3 0.0 8.7 0.0 6.5 1 88 c9t5000CCA02A1342CDd0 2.0 1359.5 1.0 168969.8 0.0 8.7 0.0 6.4 1 90 c9t5000CCA02A13500Dd0 0.0 1358.5 0.0 168714.0 0.0 8.7 0.0 6.4 1 90 c9t5000CCA02A13316Dd0 0.0 1345.5 0.0 166669.3 0.0 9.0 0.0 6.7 1 92 c9t5000CCA02A102181d0 1.0 1317.5 1.0 164456.9 0.0 8.5 0.0 6.5 1 88 c9t5000CCA02A004271d0 4.0 1342.5 2.0 166282.2 0.0 8.5 0.0 6.3 1 88 c9t5000CCA02A1251E5d0 0.0 1377.5 0.0 170515.5 0.0 8.7 0.0 6.3 1 90 c9t5000CCA02A138899d0

Now read
# dd if=/test/zero of=/dev/null  bs=1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 13.5681 s, 158 MB/s

iostat when reading
     r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
106.0 0.0 11417.4 0.0 0.0 0.2 0.0 2.4 0 14 c9t5000CCA02A004271d0 80.0 0.0 10239.9 0.0 0.0 0.2 0.0 2.4 0 10 c9t5000CCA02A1251E5d0 110.0 0.0 12182.4 0.0 0.0 0.1 0.0 1.3 0 9 c9t5000CCA02A138899d0 102.0 0.0 11664.4 0.0 0.0 0.2 0.0 1.8 0 15 c9t5000CCA02A005699d0 99.0 0.0 10900.9 0.0 0.0 0.3 0.0 3.0 0 16 c9t5000CCA02A004229d0 107.0 0.0 11545.4 0.0 0.0 0.2 0.0 1.9 0 13 c9t5000CCA02A1151DDd0 81.0 0.0 10367.9 0.0 0.0 0.2 0.0 2.2 0 11 c9t5000CCA02A1342CDd0

Unexpected low speed! Note the busy column. When writing it about 90%, when reading it about 15%

Individual disks raw read speed (don't be confused with name change. i connect JBOD to another HBA channel)

# dd if=/dev/dsk/c8t5000CCA02A13889Ad0 of=/dev/null bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 10.9685 s, 191 MB/s
# dd if=/dev/dsk/c8t5000CCA02A1342CEd0 of=/dev/null bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 10.8024 s, 194 MB/s

The 10-disks mirror zpool read slower than a single disk.

There is no tuning in /etc/system

I tried test with FreeBSD 8.3 live CD. Reads was the same (about 150Mb/s). Also i tried SmartOS, but it can't see disks behind LSI 9205-8e controller.

For compare this is speed from SATA pool (it consist of 4 6-disk raidz2 vdev)
#dd if=CentOS-6.2-x86_64-bin-DVD1.iso of=/dev/null bs=1M
4218+1 records in
4218+1 records out
4423129088 bytes (4.4 GB) copied, 4.76552 s, 928 MB/s

     r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
  13614.4    0.0 800338.5    0.0  0.1 36.0    0.0    2.6   0 914 c6
459.9 0.0 25761.4 0.0 0.0 0.8 0.0 1.8 0 22 c6t5000CCA369D16860d0 84.0 0.0 2785.2 0.0 0.0 0.2 0.0 3.0 0 13 c6t5000CCA369D1B1E0d0 836.9 0.0 50089.5 0.0 0.0 2.6 0.0 3.1 0 60 c6t5000CCA369D1B302d0 411.0 0.0 24492.6 0.0 0.0 0.8 0.0 2.1 0 25 c6t5000CCA369D16982d0 821.9 0.0 49385.1 0.0 0.0 3.0 0.0 3.7 0 67 c6t5000CCA369CFBDA3d0 231.0 0.0 12292.5 0.0 0.0 0.5 0.0 2.3 0 18 c6t5000CCA369D17E73d0 803.9 0.0 50091.5 0.0 0.0 2.9 0.0 3.6 1 69 c6t5000CCA369D0EA93d0

PS. Before testing i flash last firmware and bios to LSI 9205-8e. It come with factory 9 version. I flashed version 13.5.
Now I think that it was not worth such a hurry.
Then i downgrade it to version 12. Read speed remains the same.
Now controllers versions

 # ./sas2flash -listall
 LSI Corporation SAS2 Flash Utility
 Version 12.00.00.00 (2011.11.08)
 Copyright (c) 2008-2011 LSI Corporation. All rights reserved

         Adapter Selected is a LSI SAS: SAS2008(B1)

Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr

----------------------------------------------------------------------------

0 SAS2008(B1) 12.00.00.00 0c.00.00.04 07.23.01.00 00:05:00:00 1 SAS2308_2(B0) 12.00.00.00 0c.00.00.04 07.23.01.00 00:84:00:00

Any suggestions or thoughts ?


_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Reply via email to