Re: [ceph-users] RBD shared between ceph clients

2016-09-23 Thread David Turner
You cannot use an RBD by multiple clients. CephFS is more likely what you are thinking of if you are looking for a Ceph solution without NFS or other software. [cid:imagedeabec.JPG@1d7dc945.458e0867] David Turner | Cloud

[ceph-users] RBD shared between ceph clients

2016-09-23 Thread mayqui . quintana
Is possible use a same rbd with two ceph clients, without install others NFS software ?? and howto ? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Faulting MDS clients, HEALTH_OK

2016-09-23 Thread Gregory Farnum
On Wed, Sep 21, 2016 at 6:24 PM, Heller, Chris wrote: > What is the interesting value in ‘session ls’? Is it ‘num_leases’ or > ‘num_caps’ leases appears to be, on average, 1. But caps seems to be 16385 > for many many clients! Yeah, it's the num_caps. Interestingly, the

Re: [ceph-users] rbd pool:replica size choose: 2 vs 3

2016-09-23 Thread Götz Reinicke - IT Koordinator
Hi, Am 23.09.16 um 05:55 schrieb Zhongyan Gu: > Hi there, > the default rbd pool replica size is 3. However, I found that in our > all ssd environment, capacity become a cost issue. We want to save > more capacity. So one option is change the replica size from 3 to 2. > anyone can share the

[ceph-users] RGW multisite replication failures

2016-09-23 Thread Ben Morrice
Hello all, I have two separate ceph (10.2.2) clusters and have configured multisite replication between the two. I can see some buckets get synced, however others do not. Both clusters are RHEL7, and I have upgraded libcurl from 7.29 to 7.50 (to avoid http://tracker.ceph.com/issues/15915).

Re: [ceph-users] rbd pool:replica size choose: 2 vs 3

2016-09-23 Thread Wido den Hollander
> Op 23 september 2016 om 10:04 schreef mj : > > > Hi, > > On 09/23/2016 09:41 AM, Dan van der Ster wrote: > >> If you care about your data you run with size = 3 and min_size = 2. > >> > >> Wido > > We're currently running with min_size 1. Can we simply change this, >

Re: [ceph-users] rbd pool:replica size choose: 2 vs 3

2016-09-23 Thread Dan van der Ster
On Fri, Sep 23, 2016 at 9:29 AM, Wido den Hollander wrote: > > > > Op 23 september 2016 om 9:11 schreef Tomasz Kuzemko > > : > > > > > > Hi, > > > > biggest issue with replica size 2 is that if you find an inconsistent > > object you will not be able

Re: [ceph-users] rbd pool:replica size choose: 2 vs 3

2016-09-23 Thread Tomasz Kuzemko
Hi, biggest issue with replica size 2 is that if you find an inconsistent object you will not be able to tell which copy is the correct one. With replica size 3 you could assume that those 2 copies that are the same are correct. Until Ceph guarantees stored data integrity (that is - until we

Re: [ceph-users] rbd pool:replica size choose: 2 vs 3

2016-09-23 Thread Wido den Hollander
> Op 23 september 2016 om 9:11 schreef Tomasz Kuzemko > : > > > Hi, > > biggest issue with replica size 2 is that if you find an inconsistent > object you will not be able to tell which copy is the correct one. With > replica size 3 you could assume that those 2

Re: [ceph-users] rbd pool:replica size choose: 2 vs 3

2016-09-23 Thread mj
Hi, On 09/23/2016 09:41 AM, Dan van der Ster wrote: If you care about your data you run with size = 3 and min_size = 2. Wido We're currently running with min_size 1. Can we simply change this, online, with: ceph osd pool set vm-storage min_size 2 and expect everything to continue

Re: [ceph-users] Ceph repo is broken, no repodata at all

2016-09-23 Thread Wido den Hollander
> Op 23 september 2016 om 5:59 schreef Chengwei Yang > : > > > Hi list, > > I found that ceph repo is broken these days, no any repodata in the repo at > all. > > http://us-east.ceph.com/rpm-jewel/el7/x86_64/repodata/ > > it's just empty, so how can I install

Re: [ceph-users] rbd pool:replica size choose: 2 vs 3

2016-09-23 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Ja. > C.A. > Sent: 23 September 2016 09:50 > To: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] rbd pool:replica size choose: 2 vs 3 > > Hi > > with rep_size=2 and min_size=2, what

Re: [ceph-users] Snap delete performance impact

2016-09-23 Thread Adrian Saul
I am also seeing if reducing filestore queue ops limit from 500 to 250. On my graphs I can see the file store ops queue goes from 1 or 2 to 500 for the period of the load. I am looking to see if throttling down helps spread out the load. The normal ops load is not enough to worry the current

Re: [ceph-users] Snap delete performance impact

2016-09-23 Thread Nick Fisk
Looking back through my graphs when this happened to me I can see that the queue on the disks was up as high as 30 during the period when the snapshot was removed, this would explain the high latencies, the disk is literally having fits trying to jump all over the place. I need to test with

Re: [ceph-users] rbd pool:replica size choose: 2 vs 3

2016-09-23 Thread Ja. C.A.
ummmok. and, how would the affected PG recover, just replacing the affected OSD/DISK? or would the affected PG migrate to othe OSD/DISK? thx On 23/09/16 10:56, Nick Fisk wrote: > >> -Original Message- >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Ja.

Re: [ceph-users] Snap delete performance impact

2016-09-23 Thread Adrian Saul
I did some observation today - with the reduced filestore_op_threads it seems to ride out the storm better, not ideal but better. The main issue is that for the 10 minutes from the moment the rbd snap rm command is issued, the SATA systems in my configuration load up massively on disk IO and

Re: [ceph-users] rbd pool:replica size choose: 2 vs 3

2016-09-23 Thread Ja. C.A.
Hi with rep_size=2 and min_size=2, what drawbacks are removed compared with rep_size=2 and min_size=1? thx J. On 23/09/16 10:07, Wido den Hollander wrote: >> Op 23 september 2016 om 10:04 schreef mj : >> >> >> Hi, >> >> On 09/23/2016 09:41 AM, Dan van der Ster wrote:

Re: [ceph-users] Ceph on different OS version

2016-09-23 Thread Jaroslaw Owsiewski
2016-09-22 16:20 GMT+02:00 Wido den Hollander : > > > Op 22 september 2016 om 16:13 schreef Matteo Dacrema >: > > > > > > To be more precise, the node with different OS are only the OSD nodes. > > > > I haven't seen real issues, but a few which I could think of

Re: [ceph-users] Snap delete performance impact

2016-09-23 Thread Nick Fisk
Classic buffer bloat. The deletion process is probably going as fast as it can until the filestore queues fill up, only then will it start to back off. Problem is that with a queue of 500 ops, any disk is going to busy for thousands of milliseconds trying to empty it. Shorter queues may run

Re: [ceph-users] rbd pool:replica size choose: 2 vs 3

2016-09-23 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Ja. > C.A. > Sent: 23 September 2016 10:38 > To: n...@fisk.me.uk; ceph-users@lists.ceph.com > Subject: Re: [ceph-users] rbd pool:replica size choose: 2 vs 3 > > ummmok. > > and, how

[ceph-users] High OSD to Server ratio causes udev event to timeout during system boot

2016-09-23 Thread Tyler Bishop
Hi, My systems have 56 x 6T disk, dual 12 core processors and 256gb ram. CentOS 7 x64. During boot I'm having issues with the system going into emergency mode. When starting udevd "a start job is running for dev-disk-by" the timer of 1 minute 30 seconds runs out and the system fails to

Re: [ceph-users] Upgrading 0.94.6 -> 0.94.9 saturating mon node networking

2016-09-23 Thread Tyler Bishop
Your monitors are sending the new cluster map out every time it changes. This is a known issue IIRC, I remember reading a really interesting article on it a few months ago. I think theres a slideshow from CERN that explained it. - Original Message - From: "Stillwell, Bryan J"

Re: [ceph-users] RGW multisite replication failures

2016-09-23 Thread Orit Wasserman
Hi Ben, It seems to be http://tracker.ceph.com/issues/16742. It is being backported to jewel http://tracker.ceph.com/issues/16794, you can try apply it and see if it helps you. Regards, Orit On Fri, Sep 23, 2016 at 9:21 AM, Ben Morrice wrote: > Hello all, > > I have two

[ceph-users] ceph-deploy fails to copy keyring

2016-09-23 Thread David Welch
Hi, I have problems with the ceph-deploy command failing for reasons which don't seem obvious. For instance, I was trying to add a monitor: $ ceph-deploy mon add newmonhost [Skipping some output] [newmonhost][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-newmonhost/done

Re: [ceph-users] Stat speed for objects in ceph

2016-09-23 Thread Iain Buclaw
On 20 September 2016 at 20:30, Haomai Wang wrote: > On Wed, Sep 21, 2016 at 2:26 AM, Wido den Hollander wrote: >> >>> Op 20 september 2016 om 19:27 schreef Gregory Farnum : >>> >>> >>> In librados getting a stat is basically equivalent to

Re: [ceph-users] Question on RGW MULTISITE and librados

2016-09-23 Thread Yehuda Sadeh-Weinraub
On Thu, Sep 22, 2016 at 1:52 PM, Paul Nimbley wrote: > Fairly new to ceph so please excuse any misused terminology. We’re > currently exploring the use of ceph as a replacement storage backend for an > existing application. The existing application has 2 requirements

Re: [ceph-users] Question on RGW MULTISITE and librados

2016-09-23 Thread Paul Nimbley
That’s what we inferred from reading but wanted to be sure the replication was occurring at the RGW layer and not the RADOS layer. We haven't yet had a chance to test out multisite since we only have a single test cluster set up at the moment. On the topic of rgw multisite if I can ask a few

Re: [ceph-users] [EXTERNAL] Upgrading 0.94.6 -> 0.94.9 saturating mon node networking

2016-09-23 Thread Stillwell, Bryan J
Will, This issue in the tracker has an explanation of what is going on: http://tracker.ceph.com/issues/17386 So the encoding change caused the old OSDs to start requesting full OSDMap updates instead of incremental ones. I would still like to know the purpose of changing the encoding so late