Something that is very important to keep in mind with the way that the
cache tier implementation currently works in Ceph is that cache misses
are very expensive. It's really important that your workload have a
really big hot/cold data skew otherwise it's not going to work well at
all. In your case, you are doing sequential reads which is terrible for
this because for each pass you are never re-reading the same blocks, and
by the time you get to the end of the test and restart it, the first
blocks (apparently) have already been flushed. If you increased the
size of the cache tier, you might be able to fit the whole thing in
cache which should help dramatically, but that's not easy to guarantee
outside of benchmarks.
I'm guessing you are using firefly right? To improve this behaviour,
Sage implemented a new policy in the recent development releases not to
promote reads right away. Instead, we wait to promote until there are
several reads of the same object within a certain time period. That
should dramatically help in this case. You really don't want big
sequential reads being promoted into cache since cache promotion is
expensive and the disks are really good at doing that kind of thing anyway.
On the flip side, 4MB read misses are bad, but the situation is even
worse with 4K misses. Imagine for instance that you are going to do a 4K
read from a default 4MB RBD block and that object is not in cache. In
the implementation we have in firefly, the whole 4MB object will be
promoted to the cache which will in most cases require a transfer of
that object over the network to the primary OSD for the associated PG in
the cache pool. Now depending on the replication policy, that primary
cache pool OSD will fan out and write (by default) 2 extra copies of the
data to the other OSDs in the PG, so 3 total. Now assuming your cache
tier is on SSDs with co-located journals, each one of those writes will
actually be 2 writes, one to the journal, and one to the data store.
To recap: *Any* read miss regardless if it's 4KB or 4MB means at least 1
4MB object promotion, times 3 replicas (ie 12MB over the network) times
2 for the journal writes. So 24MB of data written to the cache tier
disks, no matter if it's 4KB or 4MB. Imagine you have 200 IOPS worth of
4KB read cache misses. That's roughly 4.8GB/s of writes into the cache
tier. If you are seldomly re-reading the same blocks, performance will
be absolutely terrible. On the other hand, if you have lots of small
random reads from the same set of 4MB objects, the cache tier really can
help. How much it helps vs just doing the reads from page cache is
debatable though. There's some band between page cache and disk where
the cache tier fits in, but getting everything right is going to be tricky.
The optimal situation imho is that the cache tier only promote objects
that have a lot of small random reads hitting them, and be very
conservative about promotions, especially for new writes. I don't know
whether or not cache promotion might pollute page cache in strange ways,
but that's something we also may need to be careful about.
For more info, see the following thread:
http://www.spinics.net/lists/ceph-devel/msg20189.html
Mark
On 09/10/2014 07:51 AM, Andrei Mikhailovsky wrote:
Hello guys,
I am experimeting with cache pool and running some tests to see how
adding the cache pool improves the overall performance of our small cluster.
While doing testing I've noticed that it seems that the cache pool is
writing too much on the cache pool ssds. Not sure what the issue here,
perhaps someone could help me understand what is going on.
My test cluster is:
2 x OSD servers (Each server has: 24GB ram, 12 cores, 8 hdd osds, 2 ssds
journals, 2 ssds for cache pool, 40gbit/s infiniband network capable of
25gbit/s over ipoib). Cache pool is set to 500GB with replica of 2.
4 x host servers (128GB ram, 24 core, 40gbit/s infiniband network
capable of 12gbit/s over ipoib)
So, my test is:
Simple tests using the following command: "dd if=/dev/vda of=/dev/null
bs=4M count=2000 iflag=direct". I am concurrently starting this command
on 10 virtual machines which are running on 4 host servers. The aim is
to monitor the use of cache pool when reading the same data over and
over again.
Running the above command for the first time does what I was expecting.
The osds are doing a lot of reads, the cache pool does a lot of writes
(around 250-300MB/s per ssd disk) and no reads. The dd results for the
guest vms are poor. The results of the "ceph -w" shows consistent
performance across the time.
Running the above for the second and consequent times produces IO
patterns which I was not expecting at all. The hdd osds are not doing
much (this part I expected), the cache pool still does a lot of writes
and very little reads! The dd results have improved just a little, but
not much. The results of the "ceph -w" shows performance breaks over
time. For instance, I have a peak of throughput in the first couple of
seconds (data is probably coming from the osd server's ram at high
rate). After the peak throughput has finished, the ceph reads are done
in the following way: 2-3 seconds of activity followed by 2 seconds if
inactivity) and it keeps doing that throughout the length of the test.
So, to put the numbers in perspective, when running tests over and over
again I would get around 2000 - 3000MB/s for the first two seconds,
followed by 0MB/s for the next two seconds, followed by around
150-250MB/s over 2-3 seconds, followed by 0MB/s for 2 seconds, followed
150-250MB/s over 2-3 seconds, followed by 0MB/s over 2 secods, and the
pattern repeats until the test is done.
I kept running the dd command for about 15-20 times and observed the
same behariour. The cache pool does mainly writes (around 200MB/s per
ssd) when guest vms are reading the same data over and over again. There
is very little read IO (around 20-40MB/s). Why am I not getting high
read IO? I have expected the 80GB of data that is being read from the
vms over and over again to be firmly recognised as the hot data and kept
in the cache pool and read from it when guest vms request the data.
Instead, I mainly get writes on the cache pool ssds and I am not really
sure where these writes are coming from as my hdd osds are being pretty
idle.
From the overall tests so far, introducing the cache pool has
drastically slowed down my cluster (by as much as 50-60%).
Thanks for any help
Andrei
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
