[ceph-users] default.rgw.log contains large omap object

2019-10-14 Thread Troy Ablan
Hi folks, Mimic cluster here, RGW pool with only default zone. I have a persistent error here LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool 'default.rgw.log' Search the cluster log for 'Large omap object found' for more details. I think I've narrowed it

Re: [ceph-users] default.rgw.log contains large omap object

2019-10-14 Thread Paul Emmerich
Looks like the usage log (radosgw-admin usage show), how often do you trim it? -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Oct 14, 2019 at 11:55 PM Troy Ablan

Re: [ceph-users] problem returning mon back to cluster

2019-10-14 Thread Paul Emmerich
How big is the mon's DB? As in just the total size of the directory you copied FWIW I recently had to perform mon surgery on a 14.2.4 (or was it 14.2.2?) cluster with 8 GB mon size and I encountered no such problems while syncing a new mon which took 10 minutes or so. Paul -- Paul Emmerich

Re: [ceph-users] default.rgw.log contains large omap object

2019-10-14 Thread Troy Ablan
Yep, that's on me. I did enable it in the config originally, and I think that I thought at the time that it might be useful, but I wasn't aware of a sharding caveat owing to most of our traffic is happening on one rgw user. I think I know what I need to do to fix it now though. Thanks

Re: [ceph-users] default.rgw.log contains large omap object

2019-10-14 Thread Troy Ablan
Paul, Apparently never. Appears to (potentially) have every request from the beginning of time (late last year, in my case). In our use case, we don't really need this data (not multi-tenant), so I might simply clear it. But in the case where this were an extremely high transaction cluster

Re: [ceph-users] default.rgw.log contains large omap object

2019-10-14 Thread Paul Emmerich
Yeah, the number of shards is configurable ("rgw usage num shards"? or something). Are you sure you aren't using it? This feature is not enabled by default, someone had to explicitly set "rgw enable usage log" for you to run into this problem. Paul -- Paul Emmerich Looking for help with your

Re: [ceph-users] problem returning mon back to cluster

2019-10-14 Thread Nikola Ciprich
On Tue, Oct 15, 2019 at 06:50:31AM +0200, Nikola Ciprich wrote: > > > On Mon, Oct 14, 2019 at 11:52:55PM +0200, Paul Emmerich wrote: > > How big is the mon's DB? As in just the total size of the directory you > > copied > > > > FWIW I recently had to perform mon surgery on a 14.2.4 (or was it

[ceph-users] Crashed MDS (segfault)

2019-10-14 Thread Gustavo Tonini
Dear ceph users, we're experiencing a segfault during MDS startup (replay process) which is making our FS inaccessible. MDS log messages: Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c08f49700 1 -- 192.168.8.195:6800/3181891717 <== osd.26 192.168.8.209:6821/2419345 3

Re: [ceph-users] problem returning mon back to cluster

2019-10-14 Thread Nikola Ciprich
On Mon, Oct 14, 2019 at 11:52:55PM +0200, Paul Emmerich wrote: > How big is the mon's DB? As in just the total size of the directory you > copied > > FWIW I recently had to perform mon surgery on a 14.2.4 (or was it > 14.2.2?) cluster with 8 GB mon size and I encountered no such problems >

Re: [ceph-users] problem returning mon back to cluster

2019-10-14 Thread Harald Staub
Probably same problem here. When I try to add another MON, "ceph health" becomes mostly unresponsive. One of the existing ceph-mon processes uses 100% CPU for several minutes. Tried it on 2 test clusters (14.2.4, 3 MONs, 5 storage nodes with around 2 hdd osds each). To avoid errors like "lease

Re: [ceph-users] Pool statistics via API

2019-10-14 Thread Sinan Polat
Hi Ernesto, I just opened the Dashboard and there is no menu at the top-right. Also no "?". I have a menu at the top-left which has the following items: Cluster health, Cluster, Block and Filesystems. Running Ceph version 12.2.8-89. Kind regards, Sinan Polat > Op 11 oktober 2019 om 22:09

Re: [ceph-users] Ceph Negative Objects Number

2019-10-14 Thread Igor Fedotov
Hi Lazuardi, never seen that. Just wondering what Ceph version are you running? Thanks, Igor On 10/8/2019 3:52 PM, Lazuardi Nasution wrote: Hi, I get following weird negative objects number on tiering. Why is this happening? How to get back to normal? Best regards, [root@management-a

Re: [ceph-users] Ceph Negative Objects Number

2019-10-14 Thread Lazuardi Nasution
Hi, Any body has same problem with my case? Best regards, On Tue, Oct 8, 2019, 19:52 Lazuardi Nasution wrote: > Hi, > > I get following weird negative objects number on tiering. Why is this > happening? How to get back to normal? > > Best regards, > > [root@management-a ~]# ceph df detail >

Re: [ceph-users] Openstack VM IOPS drops dramatically during Ceph recovery

2019-10-14 Thread Robert LeBlanc
On Thu, Oct 10, 2019 at 2:23 PM huxia...@horebdata.cn wrote: > > Hi, folks, > > I have a middle-size Ceph cluster as cinder backup for openstack (queens). > Duing testing, one Ceph node went down unexpected and powered up again ca 10 > minutes later, Ceph cluster starts PG recovery. To my

Re: [ceph-users] problem returning mon back to cluster

2019-10-14 Thread Nikola Ciprich
On Mon, Oct 14, 2019 at 01:40:19PM +0200, Harald Staub wrote: > Probably same problem here. When I try to add another MON, "ceph > health" becomes mostly unresponsive. One of the existing ceph-mon > processes uses 100% CPU for several minutes. Tried it on 2 test > clusters (14.2.4, 3 MONs, 5

Re: [ceph-users] tcmu-runner: mismatched sizes for rbd image size

2019-10-14 Thread Kilian Ries
@Mike Did you have the chance to update download.ceph.com repositories for the new version? I just tested the packages from shaman in our DEV environment and it seems to fix the work - after updating the packages i was not able to reproduce the error again and tcmu-runner starts up without

Re: [ceph-users] Ceph Negative Objects Number

2019-10-14 Thread Lazuardi Nasution
Hi Igor, It is the old Jewel (v10.2.11). This case is happen after I do cache-try-flush-evict-all or cache-flush-evict-all on respected tier pool. Best regards, On Mon, Oct 14, 2019 at 7:38 PM Igor Fedotov wrote: > Hi Lazuardi, > > never seen that. Just wondering what Ceph version are you

Re: [ceph-users] problem returning mon back to cluster

2019-10-14 Thread Nikola Ciprich
On Mon, Oct 14, 2019 at 04:31:22PM +0200, Nikola Ciprich wrote: > On Mon, Oct 14, 2019 at 01:40:19PM +0200, Harald Staub wrote: > > Probably same problem here. When I try to add another MON, "ceph > > health" becomes mostly unresponsive. One of the existing ceph-mon > > processes uses 100% CPU for