Re: [ceph-users] Balancer: change from crush-compat to upmap
Hi Caspar, Did you find any information regarding the migration from crush-compat to unmap? I’m facing the same situation. Thanks! De: ceph-users En nombre de Caspar Smit Enviado el: lunes, 25 de junio de 2018 12:25 Para: ceph-users Asunto: [ceph-users] Balancer: change from crush-compat to upmap Hi All, I've been using the balancer module in crush-compat mode for quite a while now and want to switch to upmap mode since all my clients are now luminous (v12.2.5) i've reweighted the compat weight-set back to as close as the original crush weights using 'ceph osd crush reweight-compat' Before i switch to upmap i presume i need to remove the compat weight set with: ceph osd crush weight-set rm-compat Will this have any significant impact (rebalancing lots of pgs) or does this have very little effect since i already reweighted everything back close to crush default weights? Thanks in advance and kind regards, Caspar ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] MDS damaged
Le dimanche 15 juillet 2018 à 11:01 -0500, Adam Tygart a écrit : > Check out the message titled "IMPORTANT: broken luminous 12.2.6 > release in repo, do not upgrade" > > It sounds like 12.2.7 should come *soon* to fix this transparently. Thanks. I didn't notice this one. I should monitor more closely the ML. This means I'll just wait for the fix with 12.2.7 ;-) Have a nice day ! -- Nicolas Huillard ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] MDS damaged
Check out the message titled "IMPORTANT: broken luminous 12.2.6 release in repo, do not upgrade" It sounds like 12.2.7 should come *soon* to fix this transparently. -- Adam On Sun, Jul 15, 2018 at 10:28 AM, Nicolas Huillard wrote: > Hi all, > > I have the same problem here: > * during the upgrade from 12.2.5 to 12.2.6 > * I restarted all the OSD server in turn, which did not trigger any bad > thing > * a few minutes after upgrading the OSDs/MONs/MDSs/MGRs (all on the > same set of servers) and unsetting noout, I upgraded the clients, which > triggers a temporary loss of connectivity between the two datacenters > > 2018-07-15 12:49:09.851204 mon.brome mon.0 172.21.0.16:6789/0 98 : cluster > [INF] Health check cleared: OSDMAP_FLAGS (was: noout flag(s) set) > 2018-07-15 12:49:09.851286 mon.brome mon.0 172.21.0.16:6789/0 99 : cluster > [INF] Cluster is now healthy > 2018-07-15 12:56:26.446062 mon.soufre mon.5 172.22.0.20:6789/0 34 : cluster > [INF] mon.soufre calling monitor election > 2018-07-15 12:56:26.446288 mon.oxygene mon.3 172.22.0.16:6789/0 13 : cluster > [INF] mon.oxygene calling monitor election > 2018-07-15 12:56:26.522520 mon.macaret mon.6 172.30.0.3:6789/0 10 : cluster > [INF] mon.macaret calling monitor election > 2018-07-15 12:56:26.539575 mon.phosphore mon.4 172.22.0.18:6789/0 20 : > cluster [INF] mon.phosphore calling monitor election > 2018-07-15 12:56:36.485881 mon.oxygene mon.3 172.22.0.16:6789/0 14 : cluster > [INF] mon.oxygene is new leader, mons oxygene,phosphore,soufre,macaret in > quorum (ranks 3,4,5,6) > 2018-07-15 12:56:36.930096 mon.oxygene mon.3 172.22.0.16:6789/0 19 : cluster > [WRN] Health check failed: 3/7 mons down, quorum > oxygene,phosphore,soufre,macaret (MON_DOWN) > 2018-07-15 12:56:37.041888 mon.oxygene mon.3 172.22.0.16:6789/0 26 : cluster > [WRN] overall HEALTH_WARN 3/7 mons down, quorum > oxygene,phosphore,soufre,macaret > 2018-07-15 12:56:55.456239 mon.oxygene mon.3 172.22.0.16:6789/0 57 : cluster > [WRN] daemon mds.fluor is not responding, replacing it as rank 0 with standby > daemon mds.brome > 2018-07-15 12:56:55.456365 mon.oxygene mon.3 172.22.0.16:6789/0 58 : cluster > [INF] Standby daemon mds.chlore is not responding, dropping it > 2018-07-15 12:56:55.456486 mon.oxygene mon.3 172.22.0.16:6789/0 59 : cluster > [WRN] daemon mds.brome is not responding, replacing it as rank 0 with standby > daemon mds.oxygene > 2018-07-15 12:56:55.464196 mon.oxygene mon.3 172.22.0.16:6789/0 60 : cluster > [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) > 2018-07-15 12:56:55.691674 mds.oxygene mds.0 172.22.0.16:6800/4212961230 1 : > cluster [ERR] Error recovering journal 0x200: (5) Input/output error > 2018-07-15 12:56:56.645914 mon.oxygene mon.3 172.22.0.16:6789/0 64 : cluster > [ERR] Health check failed: 1 mds daemon damaged (MDS_DAMAGE) > > I have above the hint about journal 0x200. The error appears much later > in the logs : > > 2018-07-15 16:34:28.567267 osd.11 osd.11 172.22.0.20:6805/2150 21 : cluster > [ERR] 6.14 full-object read crc 0x38f8faae != expected 0xed23f8df on > 6:292cf221:::200.:head > > I tried a repair and deep-scrub on PG 6.14, with the same nil result as > Alessandro. > I can't find any other error about the MDS journal 200. on the > other OSDs so I can't check CRCs. > > I'll try the next steps taken by Alessandro, but I'm in unknown > territory... > > Le mercredi 11 juillet 2018 à 18:10 +0300, Alessandro De Salvo a > écrit : >> Hi, >> >> after the upgrade to luminous 12.2.6 today, all our MDSes have been >> marked as damaged. Trying to restart the instances only result in >> standby MDSes. We currently have 2 filesystems active and 2 MDSes >> each. >> >> I found the following error messages in the mon: >> >> >> mds.0 :6800/2412911269 down:damaged >> mds.1 :6800/830539001 down:damaged >> mds.0 :6800/4080298733 down:damaged >> >> >> Whenever I try to force the repaired state with ceph mds repaired >> : I get something like this in the MDS logs: >> >> >> 2018-07-11 13:20:41.597970 7ff7e010e700 0 mds.1.journaler.mdlog(ro) >> error getting journal off disk >> 2018-07-11 13:20:41.598173 7ff7df90d700 -1 log_channel(cluster) log >> [ERR] : Error recovering journal 0x201: (5) Input/output error >> >> >> Any attempt of running the journal export results in errors, like >> this one: >> >> >> cephfs-journal-tool --rank=cephfs:0 journal export backup.bin >> Error ((5) Input/output error)2018-07-11 17:01:30.631571 7f94354fff00 >> -1 >> Header 200. is unreadable >> >> 2018-07-11 17:01:30.631584 7f94354fff00 -1 journal_export: Journal >> not >> readable, attempt object-by-object dump with `rados` >> >> >> Same happens for recover_dentries >> >> cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary >> Events by type:2018-07-11 17:04:19.770779 7f05429fef00 -1 Header >> 200. is unreadable >> Errors: >> 0 >> >> Is there something I could try to do to have the cluster back? >> >> I was able t
Re: [ceph-users] MDS damaged
Hi all, I have the same problem here: * during the upgrade from 12.2.5 to 12.2.6 * I restarted all the OSD server in turn, which did not trigger any bad thing * a few minutes after upgrading the OSDs/MONs/MDSs/MGRs (all on the same set of servers) and unsetting noout, I upgraded the clients, which triggers a temporary loss of connectivity between the two datacenters 2018-07-15 12:49:09.851204 mon.brome mon.0 172.21.0.16:6789/0 98 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noout flag(s) set) 2018-07-15 12:49:09.851286 mon.brome mon.0 172.21.0.16:6789/0 99 : cluster [INF] Cluster is now healthy 2018-07-15 12:56:26.446062 mon.soufre mon.5 172.22.0.20:6789/0 34 : cluster [INF] mon.soufre calling monitor election 2018-07-15 12:56:26.446288 mon.oxygene mon.3 172.22.0.16:6789/0 13 : cluster [INF] mon.oxygene calling monitor election 2018-07-15 12:56:26.522520 mon.macaret mon.6 172.30.0.3:6789/0 10 : cluster [INF] mon.macaret calling monitor election 2018-07-15 12:56:26.539575 mon.phosphore mon.4 172.22.0.18:6789/0 20 : cluster [INF] mon.phosphore calling monitor election 2018-07-15 12:56:36.485881 mon.oxygene mon.3 172.22.0.16:6789/0 14 : cluster [INF] mon.oxygene is new leader, mons oxygene,phosphore,soufre,macaret in quorum (ranks 3,4,5,6) 2018-07-15 12:56:36.930096 mon.oxygene mon.3 172.22.0.16:6789/0 19 : cluster [WRN] Health check failed: 3/7 mons down, quorum oxygene,phosphore,soufre,macaret (MON_DOWN) 2018-07-15 12:56:37.041888 mon.oxygene mon.3 172.22.0.16:6789/0 26 : cluster [WRN] overall HEALTH_WARN 3/7 mons down, quorum oxygene,phosphore,soufre,macaret 2018-07-15 12:56:55.456239 mon.oxygene mon.3 172.22.0.16:6789/0 57 : cluster [WRN] daemon mds.fluor is not responding, replacing it as rank 0 with standby daemon mds.brome 2018-07-15 12:56:55.456365 mon.oxygene mon.3 172.22.0.16:6789/0 58 : cluster [INF] Standby daemon mds.chlore is not responding, dropping it 2018-07-15 12:56:55.456486 mon.oxygene mon.3 172.22.0.16:6789/0 59 : cluster [WRN] daemon mds.brome is not responding, replacing it as rank 0 with standby daemon mds.oxygene 2018-07-15 12:56:55.464196 mon.oxygene mon.3 172.22.0.16:6789/0 60 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2018-07-15 12:56:55.691674 mds.oxygene mds.0 172.22.0.16:6800/4212961230 1 : cluster [ERR] Error recovering journal 0x200: (5) Input/output error 2018-07-15 12:56:56.645914 mon.oxygene mon.3 172.22.0.16:6789/0 64 : cluster [ERR] Health check failed: 1 mds daemon damaged (MDS_DAMAGE) I have above the hint about journal 0x200. The error appears much later in the logs : 2018-07-15 16:34:28.567267 osd.11 osd.11 172.22.0.20:6805/2150 21 : cluster [ERR] 6.14 full-object read crc 0x38f8faae != expected 0xed23f8df on 6:292cf221:::200.:head I tried a repair and deep-scrub on PG 6.14, with the same nil result as Alessandro. I can't find any other error about the MDS journal 200. on the other OSDs so I can't check CRCs. I'll try the next steps taken by Alessandro, but I'm in unknown territory... Le mercredi 11 juillet 2018 à 18:10 +0300, Alessandro De Salvo a écrit : > Hi, > > after the upgrade to luminous 12.2.6 today, all our MDSes have been > marked as damaged. Trying to restart the instances only result in > standby MDSes. We currently have 2 filesystems active and 2 MDSes > each. > > I found the following error messages in the mon: > > > mds.0 :6800/2412911269 down:damaged > mds.1 :6800/830539001 down:damaged > mds.0 :6800/4080298733 down:damaged > > > Whenever I try to force the repaired state with ceph mds repaired > : I get something like this in the MDS logs: > > > 2018-07-11 13:20:41.597970 7ff7e010e700 0 mds.1.journaler.mdlog(ro) > error getting journal off disk > 2018-07-11 13:20:41.598173 7ff7df90d700 -1 log_channel(cluster) log > [ERR] : Error recovering journal 0x201: (5) Input/output error > > > Any attempt of running the journal export results in errors, like > this one: > > > cephfs-journal-tool --rank=cephfs:0 journal export backup.bin > Error ((5) Input/output error)2018-07-11 17:01:30.631571 7f94354fff00 > -1 > Header 200. is unreadable > > 2018-07-11 17:01:30.631584 7f94354fff00 -1 journal_export: Journal > not > readable, attempt object-by-object dump with `rados` > > > Same happens for recover_dentries > > cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary > Events by type:2018-07-11 17:04:19.770779 7f05429fef00 -1 Header > 200. is unreadable > Errors: > 0 > > Is there something I could try to do to have the cluster back? > > I was able to dump the contents of the metadata pool with rados > export > -p cephfs_metadata and I'm currently trying the procedure > described in > http://docs.ceph.com/docs/master/cephfs/disaster-recovery-experts/#us > ing-an-alternate-metadata-pool-for-recovery > but I'm not sure if it will work as it's apparently doing nothing at > the > moment (maybe it's just very slow). > >
Re: [ceph-users] RBD image repurpose between iSCSI and QEMU VM, how to do properly ?
Jason Dillaman wrote: I am doing more experiments with Ceph iSCSI gateway and I am a bit confused on how to properly repurpose an RBD image from iSCSI target into QEMU virtual disk and back This isn't really a use case that we support nor intend to support. Your best bet would be to use an initiator in your linux host to connect to the same LUN as is being exported over iSCSI (just make sure the NTFS file system is quiesced / frozen. After some more tries I found a way which is convenient enough to me : for every step which requires changing RBD role from qemu/librbd to iscsi or back, I create an RBD snap and a clone of this snap under some new name and assign it to qemu/librbd or to gwcli/iscsi accordingly. Then I can easily drop original RBDs as they are unneeded. New Mimic functionality which frees me up from mandatory snap protection is of great help. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] chkdsk /b fails on Ceph iSCSI volume
Hi, I cloned a NTFS with bad blocks from USB HDD onto Ceph RBD volume (using ntfsclone, so the copy has sparse regions), and decided to clean bad blocks within the copy. I run chkdsk /b from WIndows and it fails on free space verification (step 5 of 5). In tcmu-runner.log I see that command 8f (SCSI Verify) is not supported. Does it mean that I should not try to run chkdsk /b on this volume at all ? (it seems that bad blocks were re-verified and cleared) Are there any plans to make user:rbd backstore support verify requests ? Thanks in advance for your replies. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] CephFS with erasure coding, do I need a cache-pool?
Dear all, we're planning a new Ceph-Clusterm, with CephFS as the main workload, and would like to use erasure coding to use the disks more efficiently. Access pattern will probably be more read- than write-heavy, on average. I don't have any practical experience with erasure- coded pools so far. I'd be glad for any hints / recommendations regarding these questions: * Is an SSD cache pool recommended/necessary for CephFS on an erasure-coded HDD pool (using Ceph Luminous and BlueStore)? * What are good values for k/m for erasure coding in practice (assuming a cluster of about 300 OSDs), to make things robust and ease maintenance (ability to take a few nodes down)? Is k/m = 6/3 a good choice? * Will it be sufficient to have k+m racks, resp. failure domains? Cheers and thanks for any advice, Oliver ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Safe to use rados -p rbd cleanup?
hello guys, in my production cluster i've many objects like this "#> rados -p rbd ls | grep 'benchmark'" ... .. . benchmark_data_inkscope.example.net_32654_object1918 benchmark_data_server_26414_object1990 ... .. . Is it safe to run "rados -p rbd cleanup" or is there any risk for my images? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
