On Nov 27, 2007 3:20 AM, Rob Logan <[EMAIL PROTECTED]> wrote:
> hmm, looks like its not scaling past 2 vdevs... verify
> with 'iostat -xce 1' and look in %b and if there
> is "pulsing" (waiting for IO)... try running
> a copy of bonnie for every two vdevs for more throughput.
>
> nice post...
>
Looks like there are some issues with raidz2. If I had more time maybe I
could do some better statistics, but from the eye raidz2 doe seem to pulse.
A plain stripe and a 5x2 raid 10 mirror both seem perform better, still some
pulsing but steady. Obviously some race conditions with the numbers, but
good enough to eyeball.
Note I ran 9 bonnie processes for this test . I might try 20 or 30 later.
It's interesting to note the different in the b/w speeds from zpool iostat
in each test. The disks are capable of at least 50MB+ sustained, but only
seem to reach that with raid10.
What factors would be causing the slow down for raidz2 (28MB) and raid0
(40MB)? I would have expected the CPUs to be fast enough to process the bit
stream into appropriate disk ops.
# TEST 2
# ./test-tank-2-bonnie "raidz2 c0t1d0 c0t2d0 c0t3d0 c1t1d0 c1t2d0 raidz2
c0t4d0 c0t5d0 c1t3d0 c1t4d0 c1t5d0" 2x5z2-20071127.1
-bash-3.2# zpool iostat -v tank
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
tank 118G 4.42T 50 1.33K 5.00M 164M
raidz2 58.9G 2.21T 25 681 2.50M 81.8M
c0t1d0 - - 11 418 682K 27.5M
c0t2d0 - - 11 418 680K 27.5M
c0t3d0 - - 11 418 683K 27.5M
c1t1d0 - - 11 418 684K 27.5M
c1t2d0 - - 11 418 683K 27.5M
raidz2 58.9G 2.21T 25 681 2.50M 81.8M
c0t4d0 - - 11 419 684K 27.5M
c0t5d0 - - 11 418 690K 27.5M
c1t3d0 - - 11 418 685K 27.5M
c1t4d0 - - 11 418 684K 27.5M
c1t5d0 - - 11 418 685K 27.5M
---------- ----- ----- ----- ----- ----- -----
-bash-3.2# zpool iostat -v tank
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
tank 121G 4.41T 849 1014 105M 121M
raidz2 60.4G 2.21T 424 507 52.5M 60.6M
c0t1d0 - - 173 309 10.6M 20.4M
c0t2d0 - - 173 309 10.6M 20.4M
c0t3d0 - - 173 309 10.6M 20.4M
c1t1d0 - - 173 309 10.6M 20.4M
c1t2d0 - - 173 309 10.6M 20.4M
raidz2 60.4G 2.21T 424 507 52.5M 60.6M
c0t4d0 - - 173 310 10.6M 20.4M
c0t5d0 - - 173 309 10.6M 20.4M
c1t3d0 - - 173 310 10.6M 20.4M
c1t4d0 - - 173 310 10.6M 20.4M
c1t5d0 - - 172 310 10.6M 20.4M
---------- ----- ----- ----- ----- ----- -----
------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
2x5z2-20071127.1 8G 21669 23 20801 6 14911 5 32020 32 113244 14 27.5
0
2x5z2-20071127.1 8G 19963 21 19217 6 14975 5 32057 32 49618 6 57.3
0
2x5z2-20071127.1 8G 20402 21 19114 6 14125 5 31924 33 43704 5 78.0
0
2x5z2-20071127.1 8G 18433 19 19087 6 14782 5 33228 34 42659 5 88.9
0
2x5z2-20071127.1 8G 18505 19 18791 6 14384 5 34001 35 43859 5 89.1
0
2x5z2-20071127.1 8G 16585 17 18908 6 14553 5 37019 37 47060 6 96.3
0
2x5z2-20071127.1 8G 16001 17 19654 6 14321 5 38482 39 46405 6 105.1
0
2x5z2-20071127.1 8G 17430 18 19396 6 14476 5 33350 34 44951 5 104.2
0
2x5z2-20071127.1 8G 20347 21 18458 6 13423 5 35332 36 44558 5 102.6
0
# TEST3
-bash-3.2# ./test-tank-2-bonnie "c0t1d0 c0t2d0 c0t3d0 c1t1d0 c1t2d0 c0t4d0
c0t5d0 c1t3d0 c1t4d0 c1t5d0" 10d-20071127.1
# Near Start
-bash-3.2# zpool iostat -v tank
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
tank 19.0G 4.51T 0 3.17K 48.9K 390M
c0t1d0 1.90G 462G 0 327 4.64K 39.0M
c0t2d0 1.90G 462G 0 325 3.41K 39.0M
c0t3d0 1.90G 462G 0 323 4.64K 39.0M
c1t1d0 1.90G 462G 0 324 4.64K 39.0M
c1t2d0 1.90G 462G 0 324 3.41K 39.0M
c0t4d0 1.90G 462G 0 325 7.12K 39.0M
c0t5d0 1.90G 462G 0 321 3.41K 39.0M
c1t3d0 1.90G 462G 0 323 4.64K 39.0M
c1t4d0 1.90G 462G 0 324 5.88K 39.0M
c1t5d0 1.90G 462G 0 325 7.12K 39.0M
---------- ----- ----- ----- ----- ----- -----
-bash-3.2# zpool iostat -v tank
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
tank 72.1G 4.46T 1.55K 1.95K 199M 242M
c0t1d0 7.20G 457G 159 200 19.9M 24.2M
c0t2d0 7.20G 457G 159 200 19.9M 24.2M
c0t3d0 7.20G 457G 159 200 19.9M 24.2M
c1t1d0 7.20G 457G 159 200 19.9M 24.2M
c1t2d0 7.21G 457G 159 200 19.9M 24.2M
c0t4d0 7.21G 457G 159 199 19.9M 24.2M
c0t5d0 7.21G 457G 159 199 19.9M 24.2M
c1t3d0 7.21G 457G 159 199 19.9M 24.2M
c1t4d0 7.21G 457G 159 199 19.9M 24.2M
c1t5d0 7.21G 457G 159 200 19.9M 24.2M
---------- ----- ----- ----- ----- ----- -----
-bash-3.2#
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
10d-20071127.1.6 8G 36056 39 68316 20 31771 12 40508 40 329097 38 55.4
0
10d-20071127.1.3 8G 34164 37 49439 16 34202 13 36400 37 99171 13 204.3
0
10d-20071127.1.7 8G 33902 36 59934 18 32351 13 36866 37 116146 14 117.6
0
10d-20071127.1.4 8G 33199 35 43401 14 34886 13 38464 39 106549 13 266.6
0
10d-20071127.1.5 8G 30920 34 50161 16 34517 13 38167 39 102316 13 257.6
0
10d-20071127.1.9 8G 34068 37 63596 19 31127 12 38814 39 234368 26 70.4
0
10d-20071127.1.8 8G 31540 34 52270 16 33148 13 37670 38 103871 13 246.2
0
10d-20071127.1.2 8G 32330 34 46459 15 33563 13 37957 38 111531 14 266.0
0
10d-20071127.1.1 8G 32766 35 54323 16 31146 12 38232 39 100640 13 236.8
0
# TEST4
-bash-3.2# ./test-tank-2-bonnie "mirror c0t1d0 c1t1d0 mirror c0t2d0 c1t2d0
mirror c0t3d0 c1t3d0
-bash-3.2# zpool iostat -v tank
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
tank 52.1G 2.21T 33 2.04K 1.85M 252M
mirror 10.4G 454G 6 418 380K 50.5M
c0t1d0 - - 1 415 194K 50.7M
c1t1d0 - - 1 415 194K 50.7M
mirror 10.4G 454G 6 418 379K 50.5M
c0t2d0 - - 1 416 194K 50.7M
c1t2d0 - - 1 416 193K 50.7M
mirror 10.4G 454G 6 418 377K 50.5M
c0t3d0 - - 1 415 192K 50.7M
c1t3d0 - - 1 415 192K 50.7M
mirror 10.4G 454G 6 417 379K 50.5M
c0t4d0 - - 1 414 191K 50.7M
c1t4d0 - - 1 415 195K 50.7M
mirror 10.4G 454G 6 417 380K 50.5M
c0t5d0 - - 1 415 194K 50.7M
c1t5d0 - - 1 415 195K 50.7M
---------- ----- ----- ----- ----- ----- -----
zpool iostat -v tank
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
tank 650K 2.27T 1.36K 1.38K 172M 166M
mirror 95.5K 464G 279 284 34.3M 33.3M
c0t1d0 - - 137 279 17.2M 33.4M
c1t1d0 - - 137 279 17.2M 33.4M
mirror 132K 464G 279 283 34.3M 33.3M
c0t2d0 - - 137 280 17.2M 33.4M
c1t2d0 - - 137 280 17.2M 33.4M
mirror 202K 464G 279 282 34.3M 33.3M
c0t3d0 - - 137 279 17.2M 33.4M
c1t3d0 - - 137 279 17.2M 33.4M
mirror 124K 464G 279 283 34.3M 33.3M
c0t4d0 - - 137 279 17.2M 33.4M
c1t4d0 - - 137 280 17.2M 33.4M
mirror 98K 464G 279 283 34.3M 33.3M
c0t5d0 - - 137 279 17.2M 33.4M
c1t5d0 - - 137 279 17.2M 33.4M
---------- ----- ----- ----- ----- ----- -----
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
5x2d-20071127.1. 8G 33105 35 38979 11 25492 10 38662 38 237041 27 204.8
1
5x2d-20071127.1. 8G 29242 31 31791 10 23453 8 36480 37 128285 15 95.4
0
5x2d-20071127.1. 8G 31638 34 31392 10 21477 8 38090 39 104454 12 128.4
0
5x2d-20071127.1. 8G 27447 30 34858 11 20981 8 39853 40 97468 12 175.9
0
5x2d-20071127.1. 8G 28063 30 30403 10 21683 8 39855 41 93215 10 392.4
0
5x2d-20071127.1. 8G 25763 27 27666 9 22512 8 43651 44 103173 11 456.3
1
5x2d-20071127.1. 8G 24879 27 29321 9 22883 8 41523 42 102918 11 425.6
1
5x2d-20071127.1. 8G 25661 27 31445 10 23966 8 37567 38 87822 10 301.1
1
5x2d-20071127.1. 8G 25152 27 30560 10 22072 8 42129 42 100375 11 431.6
1
_______________________________________________
storage-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/storage-discuss
