Could it because all the osds in it are set reweight =0? -7 5.45695 host ceph-osd3
26 hdd 1.81898 osd.26 down 0 1.00000 27 hdd 1.81898 osd.27 down 0 1.00000 30 hdd 1.81898 osd.30 down 0 1.00000 Best, Feng On Wed, Jun 19, 2019 at 11:36 AM Pelletier, Robert <[email protected]> wrote: > Here is a fuller picture. I inherited this ceph cluster from a previous > admin whom has left the company. Although I am a linux administrator, I > have very little experience with ceph and have had to learn; definitely > still a lot to learn. I do know this crush map was made manually. To me it > does not look right and would like to reorganize it, but I am concerned > about what effects that would have with a cluster that has data on it. > > > > I would like to remove both of the osd3-shelf1 and osd3-shelf2 chassis > buckets and move them to host ceph-osd3 (I don’t see a need from separate > buckets here). The “chassis” are actually two SAS disk shelves connected to > ceph-osd3 host. > > > > However, just moving one osd causes ceph to go unhealthy with > OBJECT_MISPLACED messages and takes a while to go back into a healthy > state. I am not too sure this is a big concern, but I am wondering if there > is a recommended procedure for doing this. As I am just learning, I don’t > want to do anything that will cause data loss. > > > > > > > > > > My Tree is supposed to look like below but it keeps changing to the map > further > below. Notice the drives moving from chassis osd3-shelf1 chassis to host > ceph-osd3. Does anyone know why this may happen? > > > > I wrote a script to monitor for this and to place the osds back where they > belong if they notice the change, but this should obviously not be > necessary. I would appreciate any help with this. > > > > > > > > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > > -62 14.55199 root osd3-shelf2 > > -60 14.55199 chassis ceph-osd3-shelf2 > > 3 hdd 1.81898 osd.3 down 1.00000 1.00000 > > 40 hdd 1.81898 osd.40 down 1.00000 1.00000 > > 41 hdd 1.81898 osd.41 down 1.00000 1.00000 > > 42 hdd 1.81898 osd.42 down 1.00000 1.00000 > > 43 hdd 1.81898 osd.43 down 1.00000 1.00000 > > 44 hdd 1.81898 osd.44 down 1.00000 1.00000 > > 45 hdd 1.81898 osd.45 down 1.00000 1.00000 > > 46 hdd 1.81898 osd.46 down 1.00000 1.00000 > > -58 1.71599 root osd3-internal > > -54 1.71599 chassis ceph-osd3-internal > > 34 hdd 0.42899 osd.34 down 1.00000 1.00000 > > 35 hdd 0.42899 osd.35 down 1.00000 1.00000 > > 36 hdd 0.42899 osd.36 down 1.00000 1.00000 > > 37 hdd 0.42899 osd.37 down 1.00000 1.00000 > > -50 14.55199 root osd3-shelf1 > > -56 14.55199 chassis ceph-osd3-shelf1 > > 21 hdd 1.81898 osd.21 down 1.00000 1.00000 > > 22 hdd 1.81898 osd.22 down 1.00000 1.00000 > > 23 hdd 1.81898 osd.23 down 1.00000 1.00000 > > 24 hdd 1.81898 osd.24 down 1.00000 1.00000 > > 25 hdd 1.81898 osd.25 down 1.00000 1.00000 > > 28 hdd 1.81898 osd.28 down 1.00000 1.00000 > > 29 hdd 1.81898 osd.29 down 1.00000 1.00000 > > 31 hdd 1.81898 osd.31 down 1.00000 1.00000 > > -7 5.45695 host ceph-osd3 > > 26 hdd 1.81898 osd.26 down 0 1.00000 > > 27 hdd 1.81898 osd.27 down 0 1.00000 > > 30 hdd 1.81898 osd.30 down 0 1.00000 > > -1 47.21199 root default > > -40 23.59000 rack mainehall > > -3 23.59000 host ceph-osd1 > > 0 hdd 1.81898 osd.0 up 1.00000 1.00000 > > 1 hdd 1.81898 osd.1 up 1.00000 1.00000 > > 2 hdd 1.81898 osd.2 up 1.00000 1.00000 > > 4 hdd 1.81898 osd.4 up 1.00000 1.00000 > > 5 hdd 1.81898 osd.5 up 0.90002 1.00000 > > 6 hdd 1.81898 osd.6 up 1.00000 1.00000 > > 7 hdd 1.81898 osd.7 up 1.00000 1.00000 > > 8 hdd 1.81898 osd.8 up 1.00000 1.00000 > > 9 hdd 1.81898 osd.9 up 1.00000 1.00000 > > 10 hdd 1.81898 osd.10 up 0.95001 1.00000 > > 33 hdd 1.76099 osd.33 up 1.00000 1.00000 > > 38 hdd 3.63899 osd.38 up 1.00000 1.00000 > > -42 23.62199 rack rangleyhall > > -5 23.62199 host ceph-osd2 > > 11 hdd 1.81898 osd.11 up 1.00000 1.00000 > > 12 hdd 1.81898 osd.12 up 0.90002 1.00000 > > 13 hdd 1.81898 osd.13 up 1.00000 1.00000 > > 14 hdd 1.81898 osd.14 up 1.00000 1.00000 > > 15 hdd 1.81898 osd.15 up 1.00000 1.00000 > > 16 hdd 1.81898 osd.16 up 1.00000 1.00000 > > 17 hdd 1.81898 osd.17 up 1.00000 1.00000 > > 18 hdd 1.81898 osd.18 up 0.90002 1.00000 > > 19 hdd 1.81898 osd.19 up 1.00000 1.00000 > > 20 hdd 1.81898 osd.20 up 1.00000 1.00000 > > 32 hdd 1.79300 osd.32 up 0.95001 1.00000 > > 39 hdd 3.63899 osd.39 up 1.00000 1.00000 > > > > > > But it keeps changing to this: > > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > > -62 14.55199 root osd3-shelf2 > > -60 14.55199 chassis ceph-osd3-shelf2 > > 3 hdd 1.81898 osd.3 down 1.00000 1.00000 > > 40 hdd 1.81898 osd.40 down 1.00000 1.00000 > > 41 hdd 1.81898 osd.41 down 1.00000 1.00000 > > 42 hdd 1.81898 osd.42 down 1.00000 1.00000 > > 43 hdd 1.81898 osd.43 down 1.00000 1.00000 > > 44 hdd 1.81898 osd.44 down 1.00000 1.00000 > > 45 hdd 1.81898 osd.45 down 1.00000 1.00000 > > 46 hdd 1.81898 osd.46 down 1.00000 1.00000 > > -58 1.71599 root osd3-internal > > -54 1.71599 chassis ceph-osd3-internal > > 34 hdd 0.42899 osd.34 down 1.00000 1.00000 > > 35 hdd 0.42899 osd.35 down 1.00000 1.00000 > > 36 hdd 0.42899 osd.36 down 1.00000 1.00000 > > 37 hdd 0.42899 osd.37 down 1.00000 1.00000 > > -50 14.55199 root osd3-shelf1 > > -56 14.55199 chassis ceph-osd3-shelf1 > > 21 hdd 1.81898 osd.21 down 1.00000 1.00000 > > 22 hdd 1.81898 osd.22 down 1.00000 1.00000 > > 23 hdd 1.81898 osd.23 down 1.00000 1.00000 > > 24 hdd 1.81898 osd.24 down 1.00000 1.00000 > > 25 hdd 1.81898 osd.25 down 1.00000 1.00000 > > -7 5.45695 host ceph-osd3 > > 26 hdd 1.81898 osd.26 down 0 1.00000 > > 27 hdd 1.81898 osd.27 down 0 1.00000 > > 30 hdd 1.81898 osd.30 down 0 1.00000 > > 28 hdd 1.81898 osd.28 down 1.00000 1.00000 > > 29 hdd 1.81898 osd.29 down 1.00000 1.00000 > > 31 hdd 1.81898 osd.31 down 1.00000 1.00000 > > > > -1 47.21199 root default > > -40 23.59000 rack mainehall > > -3 23.59000 host ceph-osd1 > > 0 hdd 1.81898 osd.0 up 1.00000 1.00000 > > 1 hdd 1.81898 osd.1 up 1.00000 1.00000 > > 2 hdd 1.81898 osd.2 up 1.00000 1.00000 > > 4 hdd 1.81898 osd.4 up 1.00000 1.00000 > > 5 hdd 1.81898 osd.5 up 0.90002 1.00000 > > 6 hdd 1.81898 osd.6 up 1.00000 1.00000 > > 7 hdd 1.81898 osd.7 up 1.00000 1.00000 > > 8 hdd 1.81898 osd.8 up 1.00000 1.00000 > > 9 hdd 1.81898 osd.9 up 1.00000 1.00000 > > 10 hdd 1.81898 osd.10 up 0.95001 1.00000 > > 33 hdd 1.76099 osd.33 up 1.00000 1.00000 > > 38 hdd 3.63899 osd.38 up 1.00000 1.00000 > > -42 23.62199 rack rangleyhall > > -5 23.62199 host ceph-osd2 > > 11 hdd 1.81898 osd.11 up 1.00000 1.00000 > > 12 hdd 1.81898 osd.12 up 0.90002 1.00000 > > 13 hdd 1.81898 osd.13 up 1.00000 1.00000 > > 14 hdd 1.81898 osd.14 up 1.00000 1.00000 > > 15 hdd 1.81898 osd.15 up 1.00000 1.00000 > > 16 hdd 1.81898 osd.16 up 1.00000 1.00000 > > 17 hdd 1.81898 osd.17 up 1.00000 1.00000 > > 18 hdd 1.81898 osd.18 up 0.90002 1.00000 > > 19 hdd 1.81898 osd.19 up 1.00000 1.00000 > > 20 hdd 1.81898 osd.20 up 1.00000 1.00000 > > 32 hdd 1.79300 osd.32 up 0.95001 1.00000 > > 39 hdd 3.63899 osd.39 up 1.00000 1.00000 > > *Robert Pelletier, IT and Security Specialist* > > Eastern Maine Community College > (207) 974-4782 | 354 Hogan Rd., Bangor, ME 04401 > > > > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
