[ceph-users] Running Openstack Nova and Ceph OSD on same machine

Hi all, is anyone running nova compute on ceph OSD Servers and could share his experience? Thanks and Regards, Felix Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender

[ceph-users] BAD nvme SSD performance

Hi all, I've recently buy two Samsung SM951 256GB nvme PCIe SSDs and built a 2 OSD ceph cluster with min_size = 1. I've tested them with fio ad I obtained two very different results with these two situations with fio. This is the command : fio --ioengine=libaio --direct=1 --name=test

Re: [ceph-users] PG won't stay clean

On Mon, Oct 26, 2015 at 4:38 AM, Robert LeBlanc wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > I set debug_osd = 20/20 and restarted the primary osd. The logs are at > http://162.144.87.113/files/ceph-osd.110.log.xz . > > The PG in question is 9.e3 and it is

[ceph-users] v0.94.5 Hammer released

This Hammer point release fixes a critical regression in librbd that can cause Qemu/KVM to crash when caching is enabled on images that have been cloned. All v0.94.4 Hammer users are strongly encouraged to upgrade. Notable Changes === * librbd: potential assertion failure during

[ceph-users] copying files from one pool to another results in more free space?

Hi All I'm observing some weird behavior in the amount of space ceph reports while copying files from an rbd image in one pool to an rbd image in another. The AVAIL number reported by 'ceph df' goes up as the copy proceeds rather than goes down! The output of 'ceph df' shows

[ceph-users] Understanding the number of TCP connections between clients and OSDs

We've run into issues with the number of open TCP connections from a single client to the OSDs in our Ceph cluster. We can (& have) increased the open file limit to work around this, but we're looking to understand what determines the number of open connections maintained between a client and a

Re: [ceph-users] Understanding the number of TCP connections between clients and OSDs

If we're talking about RBD clients (qemu) then the number also grows with number of volumes attached to the client. With a single volume it was <1000. It grows when there's heavy IO happening in the guest. I had to bump up the file open limits to several thusands (8000 was it?) to accomodate

Re: [ceph-users] Understanding the number of TCP connections between clients and OSDs

Hi, I also concerns about this problem. And my problem is how many threads will the qemu-system-x86 has. From what i tested, it could between 100 to 800, yeah, maybe it has relationship with the osd number. But it seems affect the performance when it has many threads. From what i tested, 4k

Re: [ceph-users] BAD nvme SSD performance

Hi Matteo, Ceph introduces latency into the write path and so what you are seeing is typical. If you increase the iodepth of the fio test you should get higher results though, until you start maxing out your CPU. Nick From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On

Re: [ceph-users] BAD nvme SSD performance

On 26-10-15 14:29, Matteo Dacrema wrote: > Hi Nick, > > > > I also tried to increase iodepth but nothing has changed. > > > > With iostat I noticed that the disk is fully utilized and write per > seconds from iostat match fio output. > Ceph isn't fully optimized to get the maximum

Re: [ceph-users] cache tier write-back upper bound?

Hi Brian, > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Brian Kroth > Sent: 23 October 2015 21:31 > To: ceph-users@lists.ceph.com > Subject: [ceph-users] cache tier write-back upper bound? > > Hi, I'm wondering when using a cache pool

Re: [ceph-users] why was osd pool default size changed from 2 to 3.

>but because there were only two copies it had no way to tell which one was >correct, and when I forced it to choose it often chose wrong. Yeah. This is a BIG problem with only running with two copies. Good luck if your pgs ever get inconsistent :) --Lincoln > On Oct 26, 2015, at 10:41 AM,

Re: [ceph-users] BAD nvme SSD performance

Hello, On Mon, 26 Oct 2015 14:35:19 +0100 Wido den Hollander wrote: > > > On 26-10-15 14:29, Matteo Dacrema wrote: > > Hi Nick, > > > > > > > > I also tried to increase iodepth but nothing has changed. > > > > > > > > With iostat I noticed that the disk is fully utilized and write per

Re: [ceph-users] Running Openstack Nova and Ceph OSD on same machine

I am. For our workloads it works fine. The biggest trick I found is to make sure that Nova leaves enough free RAM to not starve the OSDs. In my case, each node is running three OSDs, so in my nova.cfg I added "reserved_host_memory_mb = 3072" to help ensure that. Each node has 72GB of RAM, so

Re: [ceph-users] why was osd pool default size changed from 2 to 3.

TL;DR - Running two copies in my cluster cost me a weekend, and many more hours of productive time during normal working hours. Networking problems can be just as destructive as disk problems. I only run 2 copies on throwaway data. So, I have personal experience in data loss when running only two

Re: [ceph-users] BAD nvme SSD performance

One thing, *don't* trust iostat disk util% in case of SSDs..100% doesn't mean you are saturating SSDs there..I have seen a large performance delta even if iostat is reporting 100% disk util in both the cases. Also, the ceph.conf file you are using is not optimal..Try to add these..

[ceph-users] Not possible to remove cache tier with RBDs open?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I'm trying to remove a cache tier while I have a VM running, but a few of the objects will not evict. Do you have to stop all RBD access to remove a cache tier? # ceph osd tier cache-mode ssd-pool forward # rados -p ssd-pool cache-flush-evict-all -

Re: [ceph-users] Not possible to remove cache tier with RBDs open?

I'd imagine this is a side-effect of the fact that RBD keeps a watch/notify handle open on the header. Watch/notify is what allows you to manipulate an image from the rbd cli on a running image (i.e. take a snapshot, resize, etc). Even if the OSDs permitted this action, librbd would

Re: [ceph-users] BAD nvme SSD performance

Another point, As Christian mentioned, try to evaluate O_DIRECT|O_DSYNC performance of a SSD before choosing that for Ceph.. Try to run with direct=1 and sync =1 with fio to a raw ssd drive.. Thanks & Regards Somnath -Original Message- From: ceph-users