Hi,
We're interested in getting the highest possible read performance on a
server. To that end, we have a high-end server with multiple solid state
disks (SSDs). Since BtrFS outperformed other Linux filesystem, we choose
that. Unfortunately, there seems to be an upper boundary in the
performance of BtrFS of roughly 1 GiByte/s read speed. Compare the
following results with either BTRFS on Ubuntu versus ZFS on FreeBSD:
ZFS BtrFS
1 SSD 256 MiByte/s 256 MiByte/s
2 SSDs 505 MiByte/s 504 MiByte/s
3 SSDs 736 MiByte/s 756 MiByte/s
4 SSDs 952 MiByte/s 916 MiByte/s
5 SSDs 1226 MiByte/s 986 MiByte/s
6 SSDs 1450 MiByte/s 978 MiByte/s
8 SSDs 1653 MiByte/s 932 MiByte/s
16 SSDs 2750 MiByte/s 919 MiByte/s
The results were originally measured on a Dell PowerEdge T610, but were
repeated using a SuperMicro machine with 4 independent SAS+SATA
controllers. We made sure that the PCI-e slots where not the bottleneck.
The above results were for Ubuntu 10.04.1 server, with BtrFS v0.19,
although earlier tests with Ubuntu 9.10 showed the same results.
Apparently, the limitation is not in the hardware (the same hardware
with ZFS did scale near linear). We also tested both hardware RAID-0,
software RAID-0 (md), and using the btrfs built-in software RAID-0, but
the differences were small (<10%) (md-based software RAID was marginally
slower on Linux; RAIDZ was marginally faster on FreeBSD). So we presume
that the bottleneck is somewhere in the BtrFS (or kernel) software.
Are there suggestions how to tune the read performance? We like to scale
this up to 32 solid state disks. The -o ssd option did not improve
overall performance, although it did gave more stable results (less
fluctuation in repeated tests).
Note that the write speeds did scale fine. In the scenario with 16 solid
state disks, the write speed is 1596 MiByte/s (1.7 times as fast as the
read speed! Suffice to say that for a single disk, write is much slower
than read...).
Here are the exact settings:
~# mkfs.btrfs -d raid0 /dev/sdd /dev/sde /dev/sdf /dev/sdg \
/dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm \
/dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds
nodesize 4096 leafsize 4096 sectorsize 4096 size 2.33TB
Btrfs Btrfs v0.19
~# mount -t btrfs -o ssd /dev/sdd /mnt/ssd6
~# iozone -s 32G -r 1024 -i 0 -i 1 -w -f /mnt/ssd6/iozone.tmp
KB reclen write rewrite read reread
33554432 1024 1628475 1640349 943416 951135
Regards,
Freek Dijkstra
SARA high performance- and computing
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
