Re: [ceph-users] Luminous cluster in very bad state need some assistance.
Hi, Sorry for late reaction. With the help of Sage we finally recover our cluster. How we have recover ? It seem that due to the network flaps , some pg(s) of two of our pools was not in good state. before doing thing well i tried many things i see in the list and manipulate pg without using ceph-objectstore-tool. This probably didn't help us and conduct to some lost of data. So with the pressure to come back operational situation we decided to remove one of the two pools with problematics pg. This pools was mainly used for rbd image for our internal kvm infrastructure for which we had backup for most vm. Before removing the pool, we tried to extract most images as we can. Many was completly corrupt, but for many others we were able to extract 99% of the content and a fsck at os level let us get data. After removed this pool there are still some pg in bad state for customer facing pool. The problem was that those pg(s) was blocked by osd(s) that didn't want to join again the cluser. To solve this, we created an empty osd with weight of 0.0 We were able to extract the pg from the faulty osd(s) and inject them into the freshly create osd using the import / export command of the ceph-objectstore-tool. After that the cluster completly recover but with still osd(s) that didn't want to join the cluster. But as data of those osd are not needed any more we decided that we will restart it from scratch. What we learned of this experience. - Ensure that you network is rock solid. ( ceph realy dislike very unstable network) avoid layer 2 internconnection between your DC. and have a flat layer2 network. - keep calm and first let the time to the cluster to do theire job. ( can take some time) - never manipulate pg without using ceph-objectstore-tool or you will be in trouble. - have spare disk on some node of the cluster to be able to have empty osd to make some recovery. I would like again thanks the comunity and Sage in particular to save us from a complete disaster. Kr Philippe. From: Philippe Van Hecke Sent: 04 February 2019 07:27 To: Sage Weil Cc: ceph-users@lists.ceph.com; Belnet Services; ceph-de...@vger.kernel.org Subject: Re: [ceph-users] Luminous cluster in very bad state need some assistance. Sage, Not during the network flap or before flap , but after i had already tried the ceph-objectstore-tool remove export with no possibility to do it. And conf file never had the "ignore_les" option. I was even not aware of the existence of this option and seem that it preferable to forget that it inform me about it immediately :-) Kr Philippe. On Mon, 4 Feb 2019, Sage Weil wrote: > On Mon, 4 Feb 2019, Philippe Van Hecke wrote: > > Hi Sage, First of all tanks for your help > > > > Please find here > > https://filesender.belnet.be/?s=download=dea0edda-5b6a-4284-9ea1-c1fdf88b65e9 Something caused the version number on this PG to reset, from something like 54146'56789376 to 67932'2. Was there any operator intervention in the cluster before or during the network flapping? Or did someone by chance set the (very dangerous!) ignore_les option in ceph.conf? sage ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Bluestore increased disk usage
Hi Yenya, I guess Ceph adds the size of all your data.db devices to the cluster total used space. Regards, Jakub pt., 8 lut 2019, 10:11: Jan Kasprzak napisał(a): > Hello, ceph users, > > I moved my cluster to bluestore (Ceph Mimic), and now I see the increased > disk usage. From ceph -s: > > pools: 8 pools, 3328 pgs > objects: 1.23 M objects, 4.6 TiB > usage: 23 TiB used, 444 TiB / 467 TiB avail > > I use 3-way replication of my data, so I would expect the disk usage > to be around 14 TiB. Which was true when I used filestore-based Luminous > OSDs > before. Why the disk usage now is 23 TiB? > > If I remember it correctly (a big if!), the disk usage was about the same > when I originally moved the data to empty bluestore OSDs by changing the > crush rule, but went up after I have added more bluestore OSDs and the > cluster > rebalanced itself. > > Could it be some miscalculation of free space in bluestore? Also, could it > be > related to the HEALTH_ERR backfill_toofull problem discused here in the > other > thread? > > Thanks, > > -Yenya > > -- > | Jan "Yenya" Kasprzak > | > | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 > | > This is the world we live in: the way to deal with computers is to google > the symptoms, and hope that you don't have to watch a video. --P. Zaitcev > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
