Re: [ceph-users] Live database files on Ceph
As others have mentioned, there is no reason you cannot run databases on Ceph storage. I've been running/testing Postgres and Mysql/Mariadb on Ceph RDB volumes for quite a while now - since version 0.50 (typically inside KVM containers but via the kernel driver will work too). With a reasonable number of OSD + SSD for journals pretty good performance should be possible. However you may see some asymmetry in this performance - fast random and sequential writes, fast random reads but considerably slower sequential reads. The RBD cache may help here, but I need to investigate this further (and also some of the more fiddly settings to do with vertio disk config). Regards Mark On 04/04/14 07:04, Brian Beverage wrote: I am looking at setting up a Ganeti cluster using KVM and CentOS. While looking at storage I first looked at Gluster but noticed in the documentation it does not allow Live Database files to be saved to it. Does Ceph allow the use of LIVE database files being saved to it. If so does the database perform well? We have a couple Database servers that will be virtualized. I would like to know what other Ceph users are doing with their virtual environments that contain databases. I do not want to be locked into a SAN. I also would like to do this without being locked into a proprietary VM software. That is why Ganeti and KVM was the preferred software. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Live database files on Ceph
I've seen this fast everything except sequential reads asymmetry in my own simple dd tests on RBD images but haven't really understood the cause. Could you clarify what's going on that would cause that kind of asymmetry. I've been assuming that once I get around to turning on/tuning read caching on my underlying OSD nodes the situation will improve but haven't dug into that yet. ~jpr On 04/04/2014 04:46 AM, Mark Kirkwood wrote: However you may see some asymmetry in this performance - fast random and sequential writes, fast random reads but considerably slower sequential reads. The RBD cache may help here, but I need to investigate this further (and also some of the more fiddly settings to do with vertio disk config). ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Live database files on Ceph
Hi, We're running mysql in multi-master cluster (galera), mysql standalones, postgresql, mssql and oracle db's on ceph RBD via QEMU/KVM. As someone else pointed out it is usually faster with ceph, but sometimes you'll get some odd slow reads. Latency is our biggest enemy. Oracle comes with an awesome way to capture an insane amount of perf stats and through that we can see that our avg. latency for writes are ~12ms and reads slightly higher around 15ms. In our usecase that is acceptable. If we used local [SSD] disks this would be much lower ( 1-2ms). We've also experienced once that our galera cluster went out of sync due to a very stressed cluster/network (this particular cluster is satuated every now and then - both disk and network). We had to change scheduler from cfq - deadline on most db-servers to get acceptable speeds or we encountered writes taking up to 2sec whenever lots of seq. data had to be written. I wouldn't run super high precision/performance databases on it though. Your db performance will always reflect the status of your entire cluster system. I'd say for anything not requiring extremely finetuned always-consistent access times it runs very well. At the very least if you plan to do that, I'd suggest finding some way to isolate and gurantee performance for your guests no matter how busy your cluster would be (which I don't think you can do). We run with SSD journals and SSD backends for most of our db-stuff as we found using normal platter disks as backend could cause some issues if we hit a spikey period of cluster activity (even with ssd journals). Cheers, Martin On Thu, Apr 3, 2014 at 8:04 PM, Brian Beverage bbever...@americandatanetwork.com wrote: I am looking at setting up a Ganeti cluster using KVM and CentOS. While looking at storage I first looked at Gluster but noticed in the documentation it does not allow Live Database files to be saved to it. Does Ceph allow the use of LIVE database files being saved to it. If so does the database perform well? We have a couple Database servers that will be virtualized. I would like to know what other Ceph users are doing with their virtual environments that contain databases. I do not want to be locked into a SAN. I also would like to do this without being locked into a proprietary VM software. That is why Ganeti and KVM was the preferred software. Thanks, Brian ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Live database files on Ceph
One thing you can try is tuning read_ahead_kb on the OSDs and/or the RBD volume(s) and see if that helps. On some hardware we've seen this improve sequential read performance dramatically. Another big culprit that can really hurt sequential reads is fragmentation. BTRFS is particularly bad with RBD. Small writes to the objects that store the blocks behind the scenes end up being written to new areas of the disk due to COW. XFS probably won't fragment as badly, but we've some times seen lots of extents for some files a well. It's something to keep an eye on if you have a big sequential read workload. Mark On 04/04/2014 05:00 AM, John-Paul Robinson wrote: I've seen this fast everything except sequential reads asymmetry in my own simple dd tests on RBD images but haven't really understood the cause. Could you clarify what's going on that would cause that kind of asymmetry. I've been assuming that once I get around to turning on/tuning read caching on my underlying OSD nodes the situation will improve but haven't dug into that yet. ~jpr On 04/04/2014 04:46 AM, Mark Kirkwood wrote: However you may see some asymmetry in this performance - fast random and sequential writes, fast random reads but considerably slower sequential reads. The RBD cache may help here, but I need to investigate this further (and also some of the more fiddly settings to do with vertio disk config). ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Live database files on Ceph
Take a look at ProxmoxVE. Has full support for Ceph, is supported, and uses KVM/QEMU. -Original Message- From: ceph-users-boun...@lists.ceph.com [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Brian Candler Sent: Friday, April 04, 2014 1:44 AM To: Brian Beverage; ceph-users@lists.ceph.com Subject: Re: [ceph-users] Live database files on Ceph On 03/04/2014 23:43, Brian Beverage wrote: Here is some info on what I am trying to accomplish. My goal here is to find the least expensive way to get into Virtualization and storage without the cost of a SAN and Proprietary software ... I have been tasked with taking a new start up project and basically trying to incrementally move us into a VM environment without the use of a SAN. Given those objectives, I'd suggest you also have a look at ganeti. This won't give you the hyperscale storage of ceph, nor the remote S3/block/filesystem access. What it does give you is a clustered VM manager which can configure per-VM DRBD disk replication between pairs of nodes. So you basically get compute nodes with local storage, but with the ability to live-migrate VMs to their nominated secondary node, with no SAN or shared filesystem required. It's what Google use to run their internal office infrastructure, and is very actively developed and supported. Of course, you still need to test it with your workload. If you're tuning, have a look at drbd = 8.4.3: http://blogs.linbit.com/p/469/843-random-writes-faster/ Ganeti can also manage VMs using ceph rbd backend (I haven't tried that yet). The not-yet-released ganeti 2.12 has signficantly reworked this to use KVM's direct rbd protocol access, rather than going via the kernel rbd driver. http://docs.ganeti.org/ganeti/master/html/design-ceph-ganeti-support.html So even if you do go with ceph for the storage, I think you'll still find ganeti interesting as a way to manage the lifecycle of the VMs themselves. http://www.slideshare.net/gpaterno1/comparing-iaas-vmware-vs-openstack-vs-googles-ganeti-28016375 Apologies if this is OT for this list. Regards, Brian. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Live database files on Ceph
Interesting - I had tried that on the RBD volumes only (via blockdev --setra, but I think the effect is the same as tweaking read_ahead_kb directly), however it made no difference. Unfortunately I didn't think to adjust on the OSD's too - I'll try it out. One thing that seemed to make a big difference last time I played with this was specifying: io=native in the libvirt xml for a QEMU/KVM guest (upto 50% improvement for sequential reads) - however I did see one case where it seemed to hurt the random performance, but this was on non typical hardware (workstation with WD Black disks and Crucial M4 ssds), so could be due to that. Regards Mark On 05/04/14 01:06, Mark Nelson wrote: One thing you can try is tuning read_ahead_kb on the OSDs and/or the RBD volume(s) and see if that helps. On some hardware we've seen this improve sequential read performance dramatically. Another big culprit that can really hurt sequential reads is fragmentation. BTRFS is particularly bad with RBD. Small writes to the objects that store the blocks behind the scenes end up being written to new areas of the disk due to COW. XFS probably won't fragment as badly, but we've some times seen lots of extents for some files a well. It's something to keep an eye on if you have a big sequential read workload. Mark On 04/04/2014 05:00 AM, John-Paul Robinson wrote: I've seen this fast everything except sequential reads asymmetry in my own simple dd tests on RBD images but haven't really understood the cause. Could you clarify what's going on that would cause that kind of asymmetry. I've been assuming that once I get around to turning on/tuning read caching on my underlying OSD nodes the situation will improve but haven't dug into that yet. ~jpr On 04/04/2014 04:46 AM, Mark Kirkwood wrote: However you may see some asymmetry in this performance - fast random and sequential writes, fast random reads but considerably slower sequential reads. The RBD cache may help here, but I need to investigate this further (and also some of the more fiddly settings to do with vertio disk config). ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Live database files on Ceph
Thanks for the replies. Here is some info on what I am trying to accomplish. My goal here is to find the least expensive way to get into Virtualization and storage without the cost of a SAN and Proprietary software(eg Vmware, Hyper-V). We currently run about 6 servers that host our web based applications (.net) that use hospital data to provide reporting. Our user base is less than 50 concurrent users. Some of these applications have some large queries that produce several thousand rows of data. This data is not really large but because of legacy systems and SQL that has a lot of joins in it not the most efficient but does work currently. We are currently a windows shop. Windows server 2008 R2, 20012 R2 and SQL server 2008 R2. I have been tasked with taking a new start up project and basically trying to incrementally move us into a VM environment without the use of a SAN. It appears that since Ceph will host live databases it would be a good option. I am just trying to understand exactly how it fits together so I can share this information with all involved. I have a couple more questions. While looking at the Ceph documentation I am trying to determine what the server structure looks like. It appears to me that a minimal server foot print for a Ceph implementation would be 2 servers. Server 1 would contain: OSD's, Monitor Ceph Object Gateway. Server 2 would contain: OSD's 1 monitor 1 Standby Gateway Would this be correct? If I wanted to add to the Ceph Cluster would I just need to add 1 server which would contain OSD's,1 Monitor. Then I would then get the added space that is contained on this new server? I am wanting to host a few databases on CEPH should I consider scaling up the hardware just because of hosting databases? I also assume any added hardware would need to be similar since I cannot control on which server data is stored? Any insight and help is greatly appreciated. Thanks, Brian ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
