Re: [ceph-users] rbd IO monitoring

2018-11-29 Thread Wido den Hollander
On 11/30/18 5:48 AM, Michael Green wrote: > Hello collective wisdom, > > Ceph neophyte here, running v13.2.2 (mimic). > > Question: what tools are available to monitor IO stats on RBD level? > That is, IOPS, Throughput, IOs inflight and so on? > I'm testing with FIO and want to verify

[ceph-users] rbd IO monitoring

2018-11-29 Thread Michael Green
Hello collective wisdom, Ceph neophyte here, running v13.2.2 (mimic). Question: what tools are available to monitor IO stats on RBD level? That is, IOPS, Throughput, IOs inflight and so on? I'm testing with FIO and want to verify independently the IO load on each RBD image. -- Michael Green

Re: [ceph-users] Move Instance between Different Ceph and Openstack Installation

2018-11-29 Thread Konstantin Shalygin
I need to move instance from one Openstack with Ceph to another different Openstack with Ceph installation. The instance use volume to boot with another volume attach for data. The instance has 200GB volume boot and attach with 1TB volume for data. From what I know, I need to download the

Re: [ceph-users] client failing to respond to cache pressure

2018-11-29 Thread Zhenshi Zhou
Hi The cluster recovers healthy after I set "mds_cache_memory_limit" from 4G to 8G. Zhenshi Zhou 于2018年11月30日周五 上午11:04写道: > Hi > > I used to get warning message claims client faling to respond > to cache pressure. After I switch rockdb and wal data to ssd, the > message seems disappeared. > >

[ceph-users] client failing to respond to cache pressure

2018-11-29 Thread Zhenshi Zhou
Hi I used to get warning message claims client faling to respond to cache pressure. After I switch rockdb and wal data to ssd, the message seems disappeared. However it shows again yesterday and the message looks a little different: *MDS_CLIENT_RECALL_MANY: 1 MDSs have many * *clients failing to

Re: [ceph-users] Removing orphaned radosgw bucket indexes from pool

2018-11-29 Thread Bryan Stillwell
Wido, I've been looking into this large omap objects problem on a couple of our clusters today and came across your script during my research. The script has been running for a few hours now and I'm already over 100,000 'orphaned' objects! It appears that ever since upgrading to Luminous

Re: [ceph-users] How to recover from corrupted RocksDb

2018-11-29 Thread Paul Emmerich
If this is really the last copy of important data: consider making a full raw clone of the disk before running any ceph-objectstore-tool commands on it and consider getting some professional help if you are not too familiar with the inner workings of Ceph. That being said, it's basically just:

Re: [ceph-users] How to recover from corrupted RocksDb

2018-11-29 Thread Igor Fedotov
Yeah, that may be the way. Preferably to disable compaction during this procedure though. To do that please set bluestore rocksdb options = "disable_auto_compactions=true" in [osd] section in ceph.conf Thanks, Igor On 11/29/2018 4:54 PM, Paul Emmerich wrote: does objectstore-tool still

Re: [ceph-users] How to recover from corrupted RocksDb

2018-11-29 Thread Paul Emmerich
does objectstore-tool still work? If yes: export all the PGs on the OSD with objectstore-tool and important them into a new OSD. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89

Re: [ceph-users] How to recover from corrupted RocksDb

2018-11-29 Thread Igor Fedotov
'ceph-bluestore-tool repair' checks and repairs BlueStore metadata consistency not RocksDB one. It looks like you're observing CRC mismatch during DB compaction which is probably not triggered during the repair. Good point is that it looks like Bluestore's metadata are consistent and hence

Re: [ceph-users] MGR Dashboard

2018-11-29 Thread Lenz Grimmer
Hi Ashley, On 11/29/18 11:41 AM, Ashley Merrick wrote: > Managed to fix the issue with some googling from the error above. > > There is a bug with urllib3 1.24.1 which breaks the module ordered_dict (1) Good spotting! > I rolled back to a working version "pip install urllib3==1.23" and >

Re: [ceph-users] RGW Swift metadata dropped when S3 bucket versioning enabled

2018-11-29 Thread Yehuda Sadeh-Weinraub
On Wed, Nov 28, 2018 at 10:07 AM Maxime Guyot wrote: > > Hi Florian, > > You assumed correctly, the "test" container (private) was created with the > "openstack container create test", then I am using the S3 API to > enable/disable object versioning on it. > I use the following Python snippet

Re: [ceph-users] MGR Dashboard

2018-11-29 Thread Lenz Grimmer
On 11/29/18 11:29 AM, Ashley Merrick wrote: > Yeah had a few OS updates, but not related directly to CEPH. But they seem to be the root cause of the issue you're facing. Thanks for sharing the entire log entry. > The full error log after a reboot is : > > 2018-11-29 11:24:22.494 7faf046a1700 

Re: [ceph-users] How to recover from corrupted RocksDb

2018-11-29 Thread Mario Giammarco
The only strange thing is that ceph-bluestore-tool says that repair was done, no errors are found and all is ok. I ask myself what really does that tool. Mario Il giorno gio 29 nov 2018 alle ore 11:03 Wido den Hollander ha scritto: > > > On 11/29/18 10:45 AM, Mario Giammarco wrote: > > I have

Re: [ceph-users] MGR Dashboard

2018-11-29 Thread Ashley Merrick
Managed to fix the issue with some googling from the error above. There is a bug with urllib3 1.24.1 which breaks the module ordered_dict (1) I rolled back to a working version "pip install urllib3==1.23" and restarted the mgr service and all is now working. Thanks, Ashley

Re: [ceph-users] MGR Dashboard

2018-11-29 Thread Ashley Merrick
Yeah had a few OS updates, but not related directly to CEPH. The full error log after a reboot is : 2018-11-29 11:24:22.494 7faf046a1700 1 mgr[restful] server not running: no certificate configured 2018-11-29 11:24:22.586 7faf05ee4700 -1 log_channel(cluster) log [ERR] : Unhandled exception from

Re: [ceph-users] MGR Dashboard

2018-11-29 Thread Lenz Grimmer
On 11/29/18 10:28 AM, Ashley Merrick wrote: > Sorry missed the basic info!! > > Latest Mimic 13.2.2 > > Ubuntu 18.04 Thanks. So it worked before the reboot and did not afterwards? What changed? Did you perform an OS update? Would it be possible for you to paste the entire mgr log file

Re: [ceph-users] How to recover from corrupted RocksDb

2018-11-29 Thread Wido den Hollander
On 11/29/18 10:45 AM, Mario Giammarco wrote: > I have only that copy, it is a showroom system but someone put a > production vm on it. > I have a feeling this won't be easy to fix or actually fixable: - Compaction error: Corruption: block checksum mismatch - submit_transaction error:

Re: [ceph-users] How to recover from corrupted RocksDb

2018-11-29 Thread Mario Giammarco
I have only that copy, it is a showroom system but someone put a production vm on it. Il giorno gio 29 nov 2018 alle ore 10:43 Wido den Hollander ha scritto: > > > On 11/29/18 10:28 AM, Mario Giammarco wrote: > > Hello, > > I have a ceph installation in a proxmox cluster. > > Due to a temporary

Re: [ceph-users] How to recover from corrupted RocksDb

2018-11-29 Thread Wido den Hollander
On 11/29/18 10:28 AM, Mario Giammarco wrote: > Hello, > I have a ceph installation in a proxmox cluster. > Due to a temporary hardware glitch now I get this error on osd startup > > -6> 2018-11-26 18:02:33.179327 7fa1d784be00  0 osd.0 1033 crush map > has features 1009089991638532096,

Re: [ceph-users] MGR Dashboard

2018-11-29 Thread Ashley Merrick
Hey, Sorry missed the basic info!! Latest Mimic 13.2.2 Ubuntu 18.04 ,Ashley On Thu, 29 Nov 2018 at 5:26 PM, Lenz Grimmer wrote: > Hi Ashley, > > On 11/29/18 7:16 AM, Ashley Merrick wrote: > > > After rebooting a server that hosts the MGR Dashboard I am now unable to > > get the dashboard

[ceph-users] How to recover from corrupted RocksDb

2018-11-29 Thread Mario Giammarco
Hello, I have a ceph installation in a proxmox cluster. Due to a temporary hardware glitch now I get this error on osd startup -6> 2018-11-26 18:02:33.179327 7fa1d784be00 0 osd.0 1033 crush map has > features 1009089991638532096, adjusting msgr requires for osds >-5> 2018-11-26

Re: [ceph-users] MGR Dashboard

2018-11-29 Thread Lenz Grimmer
Hi Ashley, On 11/29/18 7:16 AM, Ashley Merrick wrote: > After rebooting a server that hosts the MGR Dashboard I am now unable to > get the dashboard module to run. > > Upon restarting the mgr service I see the following : > > ImportError: No module named ordered_dict > Nov 29 07:13:14 ceph-m01

Re: [ceph-users] Degraded objects afte: ceph osd in $osd

2018-11-29 Thread Marco Gaiarin
I reply to myself. > I've added a new node, added slowly 4 new OSD, but in the meantime an > OSD (not the new, not the node to remove) died. My situation now is: > root@blackpanther:~# ceph osd df tree > ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR TYPE NAME > -1

[ceph-users] install ceph-fuse on centos5

2018-11-29 Thread Zhenshi Zhou
Hi, I have a Centos5 server with kernel version 2.6.18. Does it support to mount cephfs with ceph-fuse? Thanks ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] MGR Dashboard

2018-11-29 Thread Jos Collin
http://tracker.ceph.com/issues/19913 On 29/11/18 11:46 AM, Ashley Merrick wrote: > Hey, > > After rebooting a server that hosts the MGR Dashboard I am now unable > to get the dashboard module to run. > > Upon restarting the mgr service I see the following : > > ImportError: No module named