From: "Christian Balzer" <[email protected]> > To: [email protected] > Sent: Friday, 3 October, 2014 2:06:48 AM > Subject: Re: [ceph-users] ceph, ssds, hdds, journals and caching
> On Thu, 2 Oct 2014 21:54:54 +0100 (BST) Andrei Mikhailovsky wrote: > > Hello Cephers, > > > > I am a bit lost on the best ways of using ssd and hdds for ceph > > cluster > > which uses rbd + kvm for guest vms. > > > > At the moment I've got 2 osd servers which currently have 8 hdd > > osds > > (max 16 bays) each and 4 ssd disks. Currently, I am using 2 ssds > > for osd > > journals and I've got 2x512GB ssd spare, which are waiting to be > > utilised. I am running Ubuntu 12.04 with 3.13 kernel from Ubuntu > > 14.04 > > and the latest firefly release. > > > In case you're planning to add more HDDs to those nodes, the obvious > use > case for those SSDs would be additional journals. >From what i've seen so far, the two ssds that i currently use for journaling >are happy serving 8 osds and I do not have much load on them. Having more osds >per server might change that though, you are right. But at the moment I was >hoping to improve the read performance, especially for small block sizes, >hense I was thinking of adding the caching layer. > Also depending on your use case, a kernel newer than 3.13(which also > is > not getting any upstream updates/support) might be a good idea. Yes, indeed. I am considering the latest supported kernels from Ubuntu team > > I've tried to use ceph cache pool tier and the results were not > > good. My > > small cluster slowed down by quite a bit and i've disabled the > > cache > > tier altogether. > > > Yeah, this feature is clearly a case of "wait for the next major > release > or the one after that and try again". Anyone know if the latest 0.80.6 firefly improves the cache behaviour? I've seen a bunch of changes in the cache tiering, however, I am not sure if these are addressing the stability of the tier or its efficiency? > > My question is how would one utilise the ssds in the best manner to > > achieve a good performance boost compared to a pure hdd setup? > > Should I > > enable block level caching (likes of bcache or similar) using all > > my > > ssds and do not bother using ssd journals? Should I keep the > > journals on > > two ssds and utilse the remaining two ssds for bcache? Or is there > > a > > better alternative? > > > This has all been discussed very recently here and the results where > inconclusive at best. In some cases reads were improved, but for > writes it > was potentially worse than normal Ceph journals. > Have you monitored your storage nodes (I keep recommending atop for > this) > during a high load time? If your SSDs are becoming the bottleneck and > not > the actual disks (doubtful, but verify), more journals. I am monitoring my ceph cluster with Zabbix and I do not have a significant load on the servers at all. My biggest concern is the single thread performance of vms. From what I can see, this is the main downside of ceph. On average, I am not getting much over 35-40MB/s per thread in cold data reads. This is compared with a single hdd read performance of 150-160MB/s. Having about 1/4 of the raw device performance is a bit worring, especially compared with what i've read. I should be getting about 1/2 of the raw drive performance for a single thread, but I am not. My hope was with caching tier I can increase it. > Other than that, maybe create a 1TB (usable space) SSD pool for > guests > with special speed requirements... I am planning to do this for the database volumes, however, from what I've read so far, there are performance bottlenecks and the current stable firefly is not optimised for ssds. I've not tried it myself, but it doesn't look like having a dedicated ssd pool will bring a significant increase in performance. Has anyone tried using bcache of dm-cache with ceph? Any tips on how to integrate it? From what I've read so far, they require you to format the existing hdd, which is not feasible if you have an existing live cluster. Cheers > Christian > -- > Christian Balzer Network/Systems Engineer > [email protected] Global OnLine Japan/Fusion Communications > http://www.gol.com/ > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
