Re: [ceph-users] ceph osd pg-upmap-items not working

2019-04-08 Thread Iain Buclaw
On Thu, 4 Apr 2019 at 13:32, Dan van der Ster wrote: > > There are several more fixes queued up for v12.2.12: > > 16b7cc1bf9 osd/OSDMap: add log for better debugging > 3d2945dd6e osd/OSDMap: calc_pg_upmaps - restrict optimization to > origin pools only > ab2dbc2089 osd/OSDMap: drop local pool

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-04-08 Thread Iain Buclaw
On Mon, 8 Apr 2019 at 05:01, Matt Benjamin wrote: > > Hi Christian, > > Dynamic bucket-index sharding for multi-site setups is being worked > on, and will land in the N release cycle. > What about removing orphaned shards on the master? Is the existing tools able to work with that? On the

Re: [ceph-users] osd_memory_target exceeding on Luminous OSD BlueStore

2019-04-08 Thread Dan van der Ster
Which OS are you using? With CentOS we find that the heap is not always automatically released. (You can check the heap freelist with `ceph tell osd.0 heap stats`). As a workaround we run this hourly: ceph tell mon.* heap release ceph tell osd.* heap release ceph tell mds.* heap release -- Dan

[ceph-users] radosgw cloud sync aws s3 auth failed

2019-04-08 Thread 黄明友
hi,all I had test the cloud sync module in radosgw. ceph verion is 13.2.5 , git commit id is cbff874f9007f1869bfd3821b7e33b2a6ffd4988; when sync to a aws s3 endpoint ,get http 400 error , so I use http:// protocol ,use the tcpick tool to dump some message like this. PUT /wuxi01

Re: [ceph-users] Latency spikes in OSD's based on bluestore

2019-04-08 Thread Patrik Martinsson
Hi Anthony, Thanks for answering. >> Which SSD model and firmware are you using? Which HBA? Well, from what I can see it's basically from all our SSD's, which unfortunately varies a bit. But from the example I posted the particular disk was, SSD SATA 6.0 Gb/s/0/100/1/0/0.8.0

Re: [ceph-users] NFS-Ganesha Mounts as a Read-Only Filesystem

2019-04-08 Thread junk
Possibly the client doesn't like the server returning SecType = "none"; Maybe try SecType = "sys":? Leon L. Robinson > On 6 Apr 2019, at 12:06, > wrote: > > Hi all, > > I have recently setup a Ceph cluster and on request using CephFS (MDS > version: ceph version 13.2.5

[ceph-users] how to judge the results? - rados bench comparison

2019-04-08 Thread Lars Täuber
Hi there, i'm new to ceph and just got my first cluster running. Now i'd like to know if the performance we get is expectable. Is there a website with benchmark results somewhere where i could have a look to compare with our HW and our results? This are the results: rados bench single

Re: [ceph-users] Unable to list rbd block > images in nautilus dashboard

2019-04-08 Thread Wes Cilldhaire
It's definitely ceph-mgr that is struggling here. It uses 100% of a cpu for for several tens of seconds and reports the followinf in its log a few times before anything gets displayed Traceback (most recent call last): File "/usr/local/share/ceph/mgr/dashboard/services/exception.py", line 88,

Re: [ceph-users] Unable to list rbd block > images in nautilus dashboard

2019-04-08 Thread Ricardo Dias
Hi Wes, I just filed a bug ticket in the Ceph tracker about this: http://tracker.ceph.com/issues/39140 Will work on a solution ASAP. Thanks, Ricardo Dias On 08/04/19 15:41, Wes Cilldhaire wrote: > It's definitely ceph-mgr that is struggling here. It uses 100% of a cpu for > for several tens

Re: [ceph-users] Unable to list rbd block > images in nautilus dashboard

2019-04-08 Thread Wes Cilldhaire
Thank you - On 9 Apr, 2019, at 12:50 AM, Ricardo Dias rd...@suse.com wrote: > Hi Wes, > > I just filed a bug ticket in the Ceph tracker about this: > > http://tracker.ceph.com/issues/39140 > > Will work on a solution ASAP. > > Thanks, > Ricardo Dias > > On 08/04/19 15:41, Wes Cilldhaire

Re: [ceph-users] Ceph Replication not working

2019-04-08 Thread Jason Dillaman
The log appears to be missing all the librbd log messages. The process seems to stop at attempting to open the image from the remote cluster: 2019-04-05 12:07:29.992323 7f0f3bfff700 20 rbd::mirror::image_replayer::OpenImageRequest: 0x7f0f28018a20 send_open_image Assuming you are using the

Re: [ceph-users] Ceph Replication not working

2019-04-08 Thread Jason Dillaman
On Mon, Apr 8, 2019 at 9:47 AM Vikas Rana wrote: > > Hi Jason, > > On Prod side, we have cluster ceph and on DR side we renamed to cephdr > > Accordingly, we renamed the ceph.conf to cephdr.conf on DR side. > > This setup used to work and one day we tried to promote the DR to verify the >

Re: [ceph-users] Ceph Replication not working

2019-04-08 Thread Vikas Rana
Hi Jason, On Prod side, we have cluster ceph and on DR side we renamed to cephdr Accordingly, we renamed the ceph.conf to cephdr.conf on DR side. This setup used to work and one day we tried to promote the DR to verify the replication and since then it's been a nightmare. The resync didn’t

[ceph-users] DevConf US CFP Ends Today + Planning

2019-04-08 Thread Mike Perez
Hey everyone, The CFP for DevConf US [1] ends today! I have submitted for us to have a Ceph Foundation booth, BOF space and two presentations myself which you can find on our CFP coordination pad [2]. I'll update here if our booth is accepted and a call for help. If you're planning on attending

Re: [ceph-users] radosgw cloud sync aws s3 auth failed

2019-04-08 Thread Robin H. Johnson
On Mon, Apr 08, 2019 at 06:38:59PM +0800, 黄明友 wrote: > > hi,all > >I had test the cloud sync module in radosgw. ceph verion is >13.2.5 , git commit id is >cbff874f9007f1869bfd3821b7e33b2a6ffd4988; Reading src/rgw/rgw_rest_client.cc shows that it only generates v2

Re: [ceph-users] osd_memory_target exceeding on Luminous OSD BlueStore

2019-04-08 Thread Mark Nelson
One of the difficulties with the osd_memory_target work is that we can't tune based on the RSS memory usage of the process. Ultimately it's up to the kernel to decide to reclaim memory and especially with transparent huge pages it's tough to judge what the kernel is going to do even if memory

Re: [ceph-users] bluefs-bdev-expand experience

2019-04-08 Thread Igor Fedotov
Hi Yuri, both issues from Round 2 relate to unsupported expansion for main device. In fact it doesn't work and silently bypasses the operation in you case. Please try with a different device... Also I've just submitted a PR for mimic to indicate the bypass, will backport to Luminous once

Re: [ceph-users] Inconsistent PGs caused by omap_digest mismatch

2019-04-08 Thread Gregory Farnum
On Mon, Apr 8, 2019 at 3:19 PM Bryan Stillwell wrote: > > We have two separate RGW clusters running Luminous (12.2.8) that have started > seeing an increase in PGs going active+clean+inconsistent with the reason > being caused by an omap_digest mismatch. Both clusters are using FileStore >

[ceph-users] Inconsistent PGs caused by omap_digest mismatch

2019-04-08 Thread Bryan Stillwell
We have two separate RGW clusters running Luminous (12.2.8) that have started seeing an increase in PGs going active+clean+inconsistent with the reason being caused by an omap_digest mismatch. Both clusters are using FileStore and the inconsistent PGs are happening on the .rgw.buckets.index

Re: [ceph-users] Inconsistent PGs caused by omap_digest mismatch

2019-04-08 Thread Bryan Stillwell
> On Apr 8, 2019, at 4:38 PM, Gregory Farnum wrote: > > On Mon, Apr 8, 2019 at 3:19 PM Bryan Stillwell wrote: >> >> There doesn't appear to be any correlation between the OSDs which would >> point to a hardware issue, and since it's happening on two different >> clusters I'm wondering if

[ceph-users] Try to log the IP in the header X-Forwarded-For with radosgw behind haproxy

2019-04-08 Thread Francois Lafont
Hi @all, I'm using Ceph rados gateway installed via ceph-ansible with the Nautilus version. The radosgw are behind a haproxy which add these headers (checked via tcpdump): X-Forwarded-Proto: http X-Forwarded-For: 10.111.222.55 where 10.111.222.55 is the IP address of the client. The

Re: [ceph-users] PGs stuck in created state

2019-04-08 Thread ceph
Hello Simon, Another idea is to increase choose_total_tries. Hth Mehmet Am 7. März 2019 09:56:17 MEZ schrieb Martin Verges : >Hello, > >try restarting every osd if possible. >Upgrade to a recent ceph version. > >-- >Martin Verges >Managing director > >Mobile: +49 174 9335695 >E-Mail:

Re: [ceph-users] Try to log the IP in the header X-Forwarded-For with radosgw behind haproxy

2019-04-08 Thread Pavan Rallabhandi
Refer "rgw log http headers" under http://docs.ceph.com/docs/nautilus/radosgw/config-ref/ Or even better in the code https://github.com/ceph/ceph/pull/7639 Thanks, -Pavan. On 4/8/19, 8:32 PM, "ceph-users on behalf of Francois Lafont" wrote: Hi @all, I'm using Ceph rados