Your problem sounds like osd_crush_update_on_start. While set to the default of true, when an osd starts it tells the mons which server it is on and the mons will update the crush map to reflect it. While these osds running on the host, but placed in a custom host in the crush map... when they start they will update to show which host they are running on. You probably want to disable that in the config so that your custom crush placement is not altered.
On Mon, May 21, 2018, 6:29 PM Martin, Jeremy <[email protected]> wrote: > Hello, > > I had a ceph cluster up and running for a few months now and all has been > well and good except for today where I updated two osd nodes and well still > well, these two nodes are designated within a rack and the rack is the > failure domain so they are essentially mirrors of each. The issue came > when I updated and rebooted the third node which has internal and external > disks in a shelf and the failure domain is at the actual osd level as these > are normal off the shelf disks for low priority storage that is not mission > critical. The issue is that before the reboot the crush map look and > behaved correctly but after the reboot the crush map was changed and had to > be rebuilt to get the storage back online, all was well after the > reassignment by I need to track down why it lots it configuration. The > main differences here is that the first four disks (34-37) are supposed to > be assigned to the chassis ceph-osd3-internal (like the before) and 21-31 > assigned to chassis chassis-ceph-osd3-shelf1 > (again like the before). After the reboot everything (34-37 and 21-31) > was reassigned to the host ceph-osd3. Update was from 12.2.4 to 12.2.5. > Any thoughts? > > Jeremy > > Before > > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -58 0 root osd3-internal > -54 0 chassis ceph-osd3-internal > 34 hdd 0.42899 osd.34 up 1.00000 1.00000 > 35 hdd 0.42899 osd.35 up 1.00000 1.00000 > 36 hdd 0.42899 osd.36 up 1.00000 1.00000 > 37 hdd 0.42899 osd.37 up 1.00000 1.00000 > -50 0 root osd3-shelf1 > -56 0 chassis ceph-osd3-shelf1 > 21 hdd 1.81898 osd.21 up 1.00000 1.00000 > 22 hdd 1.81898 osd.22 up 1.00000 1.00000 > 23 hdd 1.81898 osd.23 up 1.00000 1.00000 > 24 hdd 1.81898 osd.24 up 1.00000 1.00000 > 25 hdd 1.81898 osd.25 up 1.00000 1.00000 > 26 hdd 1.81898 osd.26 up 1.00000 1.00000 > 27 hdd 1.81898 osd.27 up 1.00000 1.00000 > 28 hdd 1.81898 osd.28 up 1.00000 1.00000 > 29 hdd 1.81898 osd.29 up 1.00000 1.00000 > 30 hdd 1.81898 osd.30 up 1.00000 1.00000 > 31 hdd 1.81898 osd.31 up 1.00000 1.00000 > -7 0 host ceph-osd3 > -1 47.21199 root default > -40 23.59000 rack mainehall > -3 23.59000 host ceph-osd1 > 0 hdd 1.81898 osd.0 up 1.00000 1.00000 > Additional osd's left off for brevity > -5 23.62199 host ceph-osd2 > 11 hdd 1.81898 osd.11 up 1.00000 1.00000 > Additional osd's left off for brevity > > After > > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -58 0 root osd3-internal > -54 0 chassis ceph-osd3-internal > -50 0 root osd3-shelf1 > -56 0 chassis ceph-osd3-shelf1 > -7 0 host ceph-osd3 > 21 hdd 1.81898 osd.21 up 1.00000 1.00000 > 22 hdd 1.81898 osd.22 up 1.00000 1.00000 > 23 hdd 1.81898 osd.23 up 1.00000 1.00000 > 24 hdd 1.81898 osd.24 up 1.00000 1.00000 > 25 hdd 1.81898 osd.25 up 1.00000 1.00000 > 26 hdd 1.81898 osd.26 up 1.00000 1.00000 > 27 hdd 1.81898 osd.27 up 1.00000 1.00000 > 28 hdd 1.81898 osd.28 up 1.00000 1.00000 > 29 hdd 1.81898 osd.29 up 1.00000 1.00000 > 30 hdd 1.81898 osd.30 up 1.00000 1.00000 > 31 hdd 1.81898 osd.31 up 1.00000 1.00000 > 34 hdd 0.42899 osd.34 up 1.00000 1.00000 > 35 hdd 0.42899 osd.35 up 1.00000 1.00000 > 36 hdd 0.42899 osd.36 up 1.00000 1.00000 > 37 hdd 0.42899 osd.37 up 1.00000 1.00000 > -1 47.21199 root default > -40 23.59000 rack mainehall > -3 23.59000 host ceph-osd1 > 0 hdd 1.81898 osd.0 up 1.00000 1.00000 > Additional osd's left off for brevity > -42 23.62199 rack rangleyhall > -5 23.62199 host ceph-osd2 > 11 hdd 1.81898 osd.11 up 1.00000 1.00000 > Additional osd's left off for brevity > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
