My company is purchasing a Sunfire x4500 to run our most I/O-bound databases,
and I'd like to get some advice on configuration and tuning. We're currently
looking at:
- Solaris 10 + zfs + RAID Z
- CentOS 4 + xfs + RAID 10
- CentOS 4 + ext3 + RAID 10
but we're open to other suggestions.
>From previous message threads, it looks like some of you have achieved stellar
>performance under both Solaris 10 U2/U3 with zfs and CentOS 4.4 with xfs.
>Would those of you who posted such results please describe how you tuned the
>OS/fs to yield those figures (e.g. patches, special drivers, read-ahead,
>checksumming, write-through cache settings, etc.)?
Most of our servers currently run CentOS/RedHat, and we have little experience
with Solaris, but we're not opposed to Solaris if there's a compelling reason
to switch. For example, it sounds like zfs snapshots may have a lighter
performance penalty than LVM snapshots. We've heard that just using LVM (even
without active snapshots) imposes a maximum sequential I/O rate of around 600
MB/s (although we haven't yet reached this limit experimentally).
By the way, we've also heard that Solaris is "more stable" under heavy I/O load
than Linux. Have any of you experienced this? It's hard to put much stock in
such a blanket statement, but naturally we don't want to introduce
instabilities.
Thanks in advance for your thoughts!
For reference:
Our database cluster will be 3-6 TB in size. The Postgres installation will be
8.1 (at least initially), compiled to use 32 KB blocks (rather than 8 KB). The
workload will be predominantly OLAP. The Sunfire X4500 has 2 dual-core
Opterons, 16 GB RAM, 48 SATA disks (500 GB/disk * 48 = 24 TB raw -> 12 TB
usable under RAID 10).
So far, we've seen the X4500 deliver impressive but suboptimal results using
the out-of-the-box installation of Solaris + zfs. The Linux testing is in the
early stages (no xfs, yet), but so far it yeilds comparatively modest write
rates and very poor read and rewrite rates.
===============================
Results under Solaris with zfs:
===============================
Four concurrent writers:
% time dd if=/dev/zero of=/zpool1/test/50GB-zero1 bs=1024k count=51200 ; time
sync
% time dd if=/dev/zero of=/zpool1/test/50GB-zero2 bs=1024k count=51200 ; time
sync
% time dd if=/dev/zero of=/zpool1/test/50GB-zero3 bs=1024k count=51200 ; time
sync
% time dd if=/dev/zero of=/zpool1/test/50GB-zero4 bs=1024k count=51200 ; time
sync
Seq Write (bs = 1 MB): 128 + 122 + 131 + 124 = 505 MB/s
Four concurrent readers:
% time dd if=/zpool1/test/50GB-zero1 of=/dev/null bs=1024k
% time dd if=/zpool1/test/50GB-zero2 of=/dev/null bs=1024k
% time dd if=/zpool1/test/50GB-zero3 of=/dev/null bs=1024k
% time dd if=/zpool1/test/50GB-zero4 of=/dev/null bs=1024k
Seq Read (bs = 1 MB): 181 + 177 + 180 + 178 = 716 MB/s
One bonnie++ process:
% bonnie++ -r 16384 -s 32g:32k -f -n0 -d /zpool1/test/bonnie_scratch
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
thumper1 32G:32k 604173 98 268893 43 543389 59 519.2
3
thumper1,32G:32k,,,604173,98,268893,43,,,543389,59,519.2,3,,,,,,,,,,,,,
4 concurrent synchronized bonnie++ processes:
% bonnie++ -p4
% bonnie++ -r 16384 -s 32g:32k -y -f -n0 -d /zpool1/test/bonnie_scratch
% bonnie++ -r 16384 -s 32g:32k -y -f -n0 -d /zpool1/test/bonnie_scratch
% bonnie++ -r 16384 -s 32g:32k -y -f -n0 -d /zpool1/test/bonnie_scratch
% bonnie++ -r 16384 -s 32g:32k -y -f -n0 -d /zpool1/test/bonnie_scratch
% bonnie++ -p-1
Combined results of 4 sessions:
Seq Output: 124 + 124 + 124 + 140 = 512 MB/s
Rewrite: 93 + 94 + 93 + 96 = 376 MB/s
Seq Input: 192 + 194 + 193 + 197 = 776 MB/s
Random Seek: 327 + 327 + 335 + 332 = 1321 seeks/s
=========================================
Results under CentOS 4 with ext3 and LVM:
=========================================
% bonnie++ -s 32g:32k -f -n0 -d /large_lvm_stripe/test/bonnie_scratch
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
thumper1.rt 32G:32k 346595 94 59448 11 132471 12 479.4
2
thumper1.rtkinternal,32G:32k,,,346595,94,59448,11,,,132471,12,479.4,2,,,,,,,,,,,,,
============================
Summary of bonnie++ results:
============================
sequential sequential sequential scattered
Test case write MB/s rewrite MB/s read MB/s seeks/s
------------------------- ---------- ------------ ---------- ---------
Sol10+zfs, 1 process 604 269 543 519
Sol10+zfs, 4 processes 512 376 776 1321
Cent4+ext3+LVM, 1 process 347 59 132 479
---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
