On 04/11/2014 12:23 PM, Daniel Schwager wrote:
Hi guys,
I'm interesting in the question, which setup will be faster:
a) a pool with journals on separate SSD's, OSD's on normal disks OR
b) a caching pool with SSDs only (osd+journal) in front
of a normal pool (osd+journal on normal disks)
A normal setup could be like this:
- 5-node firefly ceph-cluster, 6x spinner OSD and 2x SSD OSD each node.
- all pools have 3 replicates, no Erasure encoding.
Let's assume,
- Ethernet and SSDs are not the bottleneck (e.g. 10GBit Ethernet and fast SSDs)
- the SSD caching pool must be flushed as soon as possible (always <80% usage)
- the caching pool size is big enough (e.g. 10% of normal pool) to handle the
existing workload.
Existing workload could mean
1) to have sometime peak load on a bunch of KVM's
(e.g. Database operations and file copy operations)
2) running fio benchmark against a rbd device (which is smaller than the
caching pool).
Maybe somebody can tell me his 2 cents opinion what's the faster/better
solution? a) or b)
Heya, here's my 2 cents:
I think it's really a complicated balancing act and totally depends on
the workload. SSD journals optimally halve the number of writes going
to the disks with out slowing them down. They may also help with write
coalescing and smoothing out short-lived OSD delays. If you have a
controller with writeback cache, you can get some of the same benefits
(though for large sequential writes, SSD journals will help more). None
of this really helps with reads (but buffercache may!)
An SSD cache tier may help with write coalescing, but the big advantage
is helping reduce the impact of small reads/writes of hot data on the
spinning disks. How much of an advantage this provides vs having lots
of buffercache is another question though, and probably depends on how
big the objects are. The cache tier isn't perfect, it adds extra
overhead as objects are moved in/out. In your scenario, it also robs
you of having SSD journals which may or may not be a big deal.
If you expect few hotspots, generally lots of writes and/or large
sequential reads, and a nice balance of sequential write performance
between the spinning disks and SSDs, you might want to just use the SSDs
for journals. If you expect mostly cold data, but lots of small IO
(especially random reads) to a subset of objects that won't easily fit
in buffer cache but could fit in an SSD cache pool, that might be a good
use case for an SSD tier. Especially if you have controllers with
writeback cache for the spinning disks.
Mark
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
