Hello.
I faced with a strange performance problem with new disk shelf.
We a using ZFS system with SATA disks for a while.
It is Supermicro SC846-E16 chassis, Supermicro X8DTH-6F motherboard with
96Gb RAM and 24 HITACHI HDS723020BLA642 SATA disks attached to onboard
LSI 2008 controller.
Pretty much satisfied with it we bought additional shelf with SAS disks
for VMs hosting. New shelf is Supermicro SC846-E26 chassis. Disks model
is HITACHI HUS156060VLS600 (15K 600Gb SAS2).
Additional controller LSI 9205-8e was installed in server and connected
with JBOD.
I connected JBOD with 2 channels and setup multi path first, but when i
noticed performance problem i disabled multi path and disconnected one
cable (for sure it is not multipath cause the problem).
Problem description follow:
Creating test pool with 5 pair of mirrors (new shelf, SAS disks)
# zpool create -o version=28 -O primarycache=none test mirror
c9t5000CCA02A138899d0 c9t5000CCA02A102181d0 mirror c9t5000CCA02A13500Dd0
c9t5000CCA02A13316Dd0 mirror c9t5000CCA02A005699d0 c9t5000CCA02A004271d0
mirror c9t5000CCA02A004229d0 c9t5000CCA02A1342CDd0 mirror
c9t5000CCA02A1251E5d0 c9t5000CCA02A1151DDd0
(primarycache=none) to disable ARC influence
Testing sequential write
# dd if=/dev/zero of=/test/zero bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 1.04272 s, 2.1 GB/s
iostat when writing look like
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 1334.6 0.0 165782.9 0.0 8.4 0.0 6.3 1 86
c9t5000CCA02A1151DDd0
0.0 1345.5 0.0 169575.3 0.0 8.7 0.0 6.5 1 88
c9t5000CCA02A1342CDd0
2.0 1359.5 1.0 168969.8 0.0 8.7 0.0 6.4 1 90
c9t5000CCA02A13500Dd0
0.0 1358.5 0.0 168714.0 0.0 8.7 0.0 6.4 1 90
c9t5000CCA02A13316Dd0
0.0 1345.5 0.0 166669.3 0.0 9.0 0.0 6.7 1 92
c9t5000CCA02A102181d0
1.0 1317.5 1.0 164456.9 0.0 8.5 0.0 6.5 1 88
c9t5000CCA02A004271d0
4.0 1342.5 2.0 166282.2 0.0 8.5 0.0 6.3 1 88
c9t5000CCA02A1251E5d0
0.0 1377.5 0.0 170515.5 0.0 8.7 0.0 6.3 1 90
c9t5000CCA02A138899d0
Now read
# dd if=/test/zero of=/dev/null bs=1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 13.5681 s, 158 MB/s
iostat when reading
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
106.0 0.0 11417.4 0.0 0.0 0.2 0.0 2.4 0 14
c9t5000CCA02A004271d0
80.0 0.0 10239.9 0.0 0.0 0.2 0.0 2.4 0 10
c9t5000CCA02A1251E5d0
110.0 0.0 12182.4 0.0 0.0 0.1 0.0 1.3 0 9
c9t5000CCA02A138899d0
102.0 0.0 11664.4 0.0 0.0 0.2 0.0 1.8 0 15
c9t5000CCA02A005699d0
99.0 0.0 10900.9 0.0 0.0 0.3 0.0 3.0 0 16
c9t5000CCA02A004229d0
107.0 0.0 11545.4 0.0 0.0 0.2 0.0 1.9 0 13
c9t5000CCA02A1151DDd0
81.0 0.0 10367.9 0.0 0.0 0.2 0.0 2.2 0 11
c9t5000CCA02A1342CDd0
Unexpected low speed! Note the busy column. When writing it about 90%,
when reading it about 15%
Individual disks raw read speed (don't be confused with name change. i
connect JBOD to another HBA channel)
# dd if=/dev/dsk/c8t5000CCA02A13889Ad0 of=/dev/null bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 10.9685 s, 191 MB/s
# dd if=/dev/dsk/c8t5000CCA02A1342CEd0 of=/dev/null bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 10.8024 s, 194 MB/s
The 10-disks mirror zpool read slower than a single disk.
There is no tuning in /etc/system
I tried test with FreeBSD 8.3 live CD. Reads was the same (about
150Mb/s). Also i tried SmartOS, but it can't see disks behind LSI
9205-8e controller.
For compare this is speed from SATA pool (it consist of 4 6-disk raidz2
vdev)
#dd if=CentOS-6.2-x86_64-bin-DVD1.iso of=/dev/null bs=1M
4218+1 records in
4218+1 records out
4423129088 bytes (4.4 GB) copied, 4.76552 s, 928 MB/s
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
13614.4 0.0 800338.5 0.0 0.1 36.0 0.0 2.6 0 914 c6
459.9 0.0 25761.4 0.0 0.0 0.8 0.0 1.8 0 22
c6t5000CCA369D16860d0
84.0 0.0 2785.2 0.0 0.0 0.2 0.0 3.0 0 13
c6t5000CCA369D1B1E0d0
836.9 0.0 50089.5 0.0 0.0 2.6 0.0 3.1 0 60
c6t5000CCA369D1B302d0
411.0 0.0 24492.6 0.0 0.0 0.8 0.0 2.1 0 25
c6t5000CCA369D16982d0
821.9 0.0 49385.1 0.0 0.0 3.0 0.0 3.7 0 67
c6t5000CCA369CFBDA3d0
231.0 0.0 12292.5 0.0 0.0 0.5 0.0 2.3 0 18
c6t5000CCA369D17E73d0
803.9 0.0 50091.5 0.0 0.0 2.9 0.0 3.6 1 69
c6t5000CCA369D0EA93d0
PS. Before testing i flash last firmware and bios to LSI 9205-8e. It
come with factory 9 version. I flashed version 13.5.
Now I think that it was not worth such a hurry.
Then i downgrade it to version 12. Read speed remains the same.
Now controllers versions
# ./sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 12.00.00.00 (2011.11.08)
Copyright (c) 2008-2011 LSI Corporation. All rights reserved
Adapter Selected is a LSI SAS: SAS2008(B1)
Num Ctlr FW Ver NVDATA x86-BIOS PCI
Addr
----------------------------------------------------------------------------
0 SAS2008(B1) 12.00.00.00 0c.00.00.04 07.23.01.00
00:05:00:00
1 SAS2308_2(B0) 12.00.00.00 0c.00.00.04 07.23.01.00
00:84:00:00
Any suggestions or thoughts ?
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
