Re: [ceph-users] Mark CephFS inode as lost

please create a ticket at http://tracker.ceph.com/projects/cephfs and upload mds log with debug_mds =10 On Tue, Jul 23, 2019 at 6:00 AM Robert LeBlanc wrote: > > We have a Luminous cluster which has filled up to 100% multiple times and > this causes an inode to be left in a bad state. Doing anyt

Re: [ceph-users] Nautilus 14.2.1 / 14.2.2 crash

Have they been stable since, or still had some crash? ,Thanks On Sat, 20 Jul 2019 10:09:08 +0800 Nigel Williams wrote On Sat, 20 Jul 2019 at 04:28, Nathan Fish wrote: On further investigation, it seems to be this bug: http://tracker.ceph.com/issu

Re: [ceph-users] reproducable rbd-nbd crashes

Hi Mike, Am 22.07.19 um 16:48 schrieb Mike Christie: > On 07/22/2019 06:00 AM, Marc Schöchlin wrote: >>> With older kernels no timeout would be set for each command by default, >>> so if you were not running that tool then you would not see the nbd >>> disconnect+io_errors+xfs issue. You would jus

Re: [ceph-users] Iscsi in the nautilus Dashboard

Hi Brent,As far as I know version 3.0 (which I assume is version 9) is the minimum required for the dashboard.I would go with the latest from Shaman; it won't break the actual iSCSI part of the setup, only maybe the iSCSI support in the dashboard. I haven't tried it myself, I'm still at version 2.

Re: [ceph-users] reproducable rbd-nbd crashes

Hi Mike, Am 22.07.19 um 17:01 schrieb Mike Christie: > On 07/19/2019 02:42 AM, Marc Schöchlin wrote: >> We have ~500 heavy load rbd-nbd devices in our xen cluster (rbd-nbd 12.2.5, >> kernel 4.4.0+10, centos clone) and ~20 high load krbd devices (kernel >> 4.15.0-45, ubuntu 16.04) - we never expe

Re: [ceph-users] Iscsi in the nautilus Dashboard

I posted to the ceph-iscsi github but Dillaman noted that 3.2 was version 10. Which means that wouldn’t solve the issue with the version 9 requirement of the current 14.2.2 nautilus. Paul noted 3.1 is “pretty broken”, soo which version is version 9? Or should I hack/patch the dashboard in 14

[ceph-users] Mark CephFS inode as lost

We have a Luminous cluster which has filled up to 100% multiple times and this causes an inode to be left in a bad state. Doing anything to these files causes the client to hang which requires evicting the client and failing over the MDS. Usually we move the parent directory out of the way and thin

Re: [ceph-users] MON crashing when upgrading from Hammer to Luminous

First link should be this one http://docs.ceph.com/docs/jewel/install/upgrading-ceph/#upgrade-procedures rather than http://docs.ceph.com/docs/mimic/install/upgrading-ceph/#upgrade-procedures

Re: [ceph-users] MON crashing when upgrading from Hammer to Luminous

Hi you’ll have to go from Hammer to Jewel then from Jewel to Luminous for a smooth upgrade. - http://docs.ceph.com/docs/mimic/install/upgrading-ceph/#upgrade-procedures - http://docs.ceph.com/docs/luminous/release-not

Re: [ceph-users] Failed to get omap key when mirroring of image is enabled

On Mon, Jul 22, 2019 at 3:26 PM Ajitha Robert wrote: > > Thanks for your reply > > 1) In scenario 1, I didnt attempt to delete the cinder volume. Please find > the cinder volume log. > http://paste.openstack.org/show/754731/ It might be better to ping Cinder folks about that one. It doesn't real

Re: [ceph-users] MON / MDS Storage Location

Hi, mon: /var/lib/ceph/mon/* mds: inside the cephfs_data and cephfs_metadata rados pools On 07/22/2019 09:25 PM, dhils...@performair.com wrote: > All; > > Where, in the filesystem, do MONs and MDSs store their data? > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technolo

[ceph-users] MON / MDS Storage Location

All; Where, in the filesystem, do MONs and MDSs store their data? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com ___ ceph-users mailing list ceph-user

[ceph-users] MON crashing when upgrading from Hammer to Luminous

Dear Everyone, First of all, guys, seriously, Thank you for Ceph. now to the problem, upgrading ceph from 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403) to 12.2.12-218-g9fd889f (9fd889fe09c652512ca78854702d5ad9bf3059bb), ceph-mon seems unable to upgrade it's database, problem is gone if i --fo

Re: [ceph-users] Failed to get omap key when mirroring of image is enabled

On Mon, Jul 22, 2019 at 10:49 AM Ajitha Robert wrote: > > No error log in rbd-mirroring except some connection timeout came once, > Scenario 1: > when I create a bootable volume of 100 GB with a glance image.Image get > downloaded and from cinder, volume log throws with "volume is busy deleting

Re: [ceph-users] which tool to use for benchmarking rgw s3, yscb or cosbench

I have had good luck with YCSB as an initial assessment of different storage systems. Typically I'll use this first when I am playing with a new system, but I like to switch to the more native tools (rados bench, cassandra-stress, etc etc) as soon as I am more comfortable. And I can definitely se

Re: [ceph-users] reproducable rbd-nbd crashes

On 07/19/2019 02:42 AM, Marc Schöchlin wrote: > We have ~500 heavy load rbd-nbd devices in our xen cluster (rbd-nbd 12.2.5, > kernel 4.4.0+10, centos clone) and ~20 high load krbd devices (kernel > 4.15.0-45, ubuntu 16.04) - we never experienced problems like this. For this setup, do you have 25

Re: [ceph-users] reproducable rbd-nbd crashes

On 07/22/2019 06:00 AM, Marc Schöchlin wrote: >> With older kernels no timeout would be set for each command by default, >> so if you were not running that tool then you would not see the nbd >> disconnect+io_errors+xfs issue. You would just see slow IOs. >> >> With newer kernels, like 4.15, nbd.ko

Re: [ceph-users] Iscsi in the nautilus Dashboard

Hi all,That was not the most recent. This is it (3.2.4): https://2.chacra.ceph.com/r/ceph-iscsi/master/8a3967698257e1b49a9d554847b84418c15da902/centos/7/flavors/default/KasparOp 22 juli 2019 om 14:01 schreef Kaspar Bosma : Hi Brent,You may want to have a look at the repos at shaman.ceph.com.The l

Re: [ceph-users] Failed to get omap key when mirroring of image is enabled

On Sun, Jul 21, 2019 at 8:25 PM Ajitha Robert wrote: > > I have a rbd mirroring setup with primary and secondary clusters as peers > and I have a pool enabled image mode.., In this i created a rbd image , > enabled with journaling. > > But whenever i enable mirroring on the image, I m getting

Re: [ceph-users] Iscsi in the nautilus Dashboard

Hi Brent,You may want to have a look at the repos at shaman.ceph.com.The latest (3.2.2) packaged version of Ceph iSCSI is located here:https://4.chacra.ceph.com/r/ceph-iscsi/master/ff5e6873c43ab6828d3f7264526100b95a7e3954/centos/7/flavors/default/noarch/You can also find related package repos for

Re: [ceph-users] New best practices for osds???

OK, I meant "it may help performance" :) the main point is that we had at least one case of data loss due to some Adaptec controller in RAID0 mode discussed recently in our ceph chat... -- With best regards, Vitaliy Filippov ___ ceph-users mailing

Re: [ceph-users] reproducable rbd-nbd crashes

Hello Mike, i attached inline comments. Am 19.07.19 um 22:20 schrieb Mike Christie: > >> We have ~500 heavy load rbd-nbd devices in our xen cluster (rbd-nbd 12.2.5, >> kernel 4.4.0+10, centos clone) and ~20 high load krbd devices (kernel >> 4.15.0-45, ubuntu 16.04) - we never experienced proble

Re: [ceph-users] New best practices for osds???

On Mon, Jul 22, 2019 at 12:52 PM Vitaliy Filippov wrote: > It helps performance, Not necessarily, I've seen several setups where disabling the cache increases performance Paul > but it can also lead to data loss if the raid > controller is crap (not flushing data correctly) > > -- > With be

Re: [ceph-users] Iscsi in the nautilus Dashboard

Version 9 is the fqdn stuff which was introduced in 3.1. Use 3.2 as 3.1 is pretty broken. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Jul 22, 2019 at 3:24

Re: [ceph-users] New best practices for osds???

It helps performance, but it can also lead to data loss if the raid controller is crap (not flushing data correctly) -- With best regards, Vitaliy Filippov ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph

Re: [ceph-users] Future of Filestore?

Linear reads, `hdparm -t /dev/vda`. Check if you have `cache=writeback` enabled in your VM options. If it's enabled but you still get 5mb/s then try to benchmark your cluster with fio -ioengine=rbd from outside a VM. Like fio -ioengine=rbd -name=test -bs=4M -iodepth=16 -rw=read -pool=rpool

Re: [ceph-users] Future of Filestore?

On 22/7/19 7:39 pm, Vitaliy Filippov wrote: > 5MB/s in what mode? Linear reads, `hdparm -t /dev/vda`. > For linear writes, that definitely means some kind of misconfiguration. > For random writes... there's a handbrake in Bluestore which makes random > writes run at half speed in HDD-only setups

Re: [ceph-users] Future of Filestore?

On 22/7/19 7:13 pm, Marc Roos wrote: > > >> Reverting back to filestore is quite a lot of work and time again. > >> Maybe see first if with some tuning of the vms you can get better > results? > > > >None of the VMs are particularly disk-intensive. There's two users > accessing the system

Re: [ceph-users] which tool to use for benchmarking rgw s3, yscb or cosbench

On 2019-07-21T23:51:41, Wei Zhao wrote: > Hi: > I found cosbench is a very convenient tool for benchmaring rgw. But > when I read papers , I found YCSB tool, > https://github.com/brianfrankcooper/YCSB/tree/master/s3 . It seems > that this is used for test cloud service , and seems a right too

Re: [ceph-users] Future of Filestore?

5MB/s in what mode? For linear writes, that definitely means some kind of misconfiguration. For random writes... there's a handbrake in Bluestore which makes random writes run at half speed in HDD-only setups :) https://github.com/ceph/ceph/pull/26909 And if you push that handbrake down y

Re: [ceph-users] Future of Filestore?

>> Reverting back to filestore is quite a lot of work and time again. >> Maybe see first if with some tuning of the vms you can get better results? > >None of the VMs are particularly disk-intensive. There's two users accessing the system over a WiFi network for email, and some HTTP/SMTP