Could you show your cache tiering configuration? Especially this three
parameters.
ceph osd pool set hot-storage cache_target_dirty_ratio 0.4
ceph osd pool set hot-storage cache_target_full_ratio 0.8
ceph osd pool set {cachepool} target_max_bytes {#bytes}
From: ceph-users [mailto:[email protected]] On Behalf Of Andrei
Mikhailovsky
Sent: Wednesday, September 10, 2014 8:51 PM
To: ceph-users
Subject: [ceph-users] Cache Pool writing too much on ssds, poor performance?
Hello guys,
I am experimeting with cache pool and running some tests to see how adding the
cache pool improves the overall performance of our small cluster.
While doing testing I've noticed that it seems that the cache pool is writing
too much on the cache pool ssds. Not sure what the issue here, perhaps someone
could help me understand what is going on.
My test cluster is:
2 x OSD servers (Each server has: 24GB ram, 12 cores, 8 hdd osds, 2 ssds
journals, 2 ssds for cache pool, 40gbit/s infiniband network capable of
25gbit/s over ipoib). Cache pool is set to 500GB with replica of 2.
4 x host servers (128GB ram, 24 core, 40gbit/s infiniband network capable of
12gbit/s over ipoib)
So, my test is:
Simple tests using the following command: "dd if=/dev/vda of=/dev/null bs=4M
count=2000 iflag=direct". I am concurrently starting this command on 10 virtual
machines which are running on 4 host servers. The aim is to monitor the use of
cache pool when reading the same data over and over again.
Running the above command for the first time does what I was expecting. The
osds are doing a lot of reads, the cache pool does a lot of writes (around
250-300MB/s per ssd disk) and no reads. The dd results for the guest vms are
poor. The results of the "ceph -w" shows consistent performance across the time.
Running the above for the second and consequent times produces IO patterns
which I was not expecting at all. The hdd osds are not doing much (this part I
expected), the cache pool still does a lot of writes and very little reads! The
dd results have improved just a little, but not much. The results of the "ceph
-w" shows performance breaks over time. For instance, I have a peak of
throughput in the first couple of seconds (data is probably coming from the osd
server's ram at high rate). After the peak throughput has finished, the ceph
reads are done in the following way: 2-3 seconds of activity followed by 2
seconds if inactivity) and it keeps doing that throughout the length of the
test. So, to put the numbers in perspective, when running tests over and over
again I would get around 2000 - 3000MB/s for the first two seconds, followed by
0MB/s for the next two seconds, followed by around 150-250MB/s over 2-3
seconds, followed by 0MB/s for 2 seconds, followed 150-250MB/s over 2-3
seconds, followed by 0MB/s over 2 secods, and the pattern repeats until the
test is done.
I kept running the dd command for about 15-20 times and observed the same
behariour. The cache pool does mainly writes (around 200MB/s per ssd) when
guest vms are reading the same data over and over again. There is very little
read IO (around 20-40MB/s). Why am I not getting high read IO? I have expected
the 80GB of data that is being read from the vms over and over again to be
firmly recognised as the hot data and kept in the cache pool and read from it
when guest vms request the data. Instead, I mainly get writes on the cache pool
ssds and I am not really sure where these writes are coming from as my hdd osds
are being pretty idle.
From the overall tests so far, introducing the cache pool has drastically
slowed down my cluster (by as much as 50-60%).
Thanks for any help
Andrei
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
