Here is a fuller picture. I inherited this ceph cluster from a previous admin whom has left the company. Although I am a linux administrator, I have very little experience with ceph and have had to learn; definitely still a lot to learn. I do know this crush map was made manually. To me it does not look right and would like to reorganize it, but I am concerned about what effects that would have with a cluster that has data on it.
I would like to remove both of the osd3-shelf1 and osd3-shelf2 chassis buckets and move them to host ceph-osd3 (I don’t see a need from separate buckets here). The “chassis” are actually two SAS disk shelves connected to ceph-osd3 host. However, just moving one osd causes ceph to go unhealthy with OBJECT_MISPLACED messages and takes a while to go back into a healthy state. I am not too sure this is a big concern, but I am wondering if there is a recommended procedure for doing this. As I am just learning, I don’t want to do anything that will cause data loss. My Tree is supposed to look like below but it keeps changing to the map further below. Notice the drives moving from chassis osd3-shelf1 chassis to host ceph-osd3. Does anyone know why this may happen? I wrote a script to monitor for this and to place the osds back where they belong if they notice the change, but this should obviously not be necessary. I would appreciate any help with this. ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -62 14.55199 root osd3-shelf2 -60 14.55199 chassis ceph-osd3-shelf2 3 hdd 1.81898 osd.3 down 1.00000 1.00000 40 hdd 1.81898 osd.40 down 1.00000 1.00000 41 hdd 1.81898 osd.41 down 1.00000 1.00000 42 hdd 1.81898 osd.42 down 1.00000 1.00000 43 hdd 1.81898 osd.43 down 1.00000 1.00000 44 hdd 1.81898 osd.44 down 1.00000 1.00000 45 hdd 1.81898 osd.45 down 1.00000 1.00000 46 hdd 1.81898 osd.46 down 1.00000 1.00000 -58 1.71599 root osd3-internal -54 1.71599 chassis ceph-osd3-internal 34 hdd 0.42899 osd.34 down 1.00000 1.00000 35 hdd 0.42899 osd.35 down 1.00000 1.00000 36 hdd 0.42899 osd.36 down 1.00000 1.00000 37 hdd 0.42899 osd.37 down 1.00000 1.00000 -50 14.55199 root osd3-shelf1 -56 14.55199 chassis ceph-osd3-shelf1 21 hdd 1.81898 osd.21 down 1.00000 1.00000 22 hdd 1.81898 osd.22 down 1.00000 1.00000 23 hdd 1.81898 osd.23 down 1.00000 1.00000 24 hdd 1.81898 osd.24 down 1.00000 1.00000 25 hdd 1.81898 osd.25 down 1.00000 1.00000 28 hdd 1.81898 osd.28 down 1.00000 1.00000 29 hdd 1.81898 osd.29 down 1.00000 1.00000 31 hdd 1.81898 osd.31 down 1.00000 1.00000 -7 5.45695 host ceph-osd3 26 hdd 1.81898 osd.26 down 0 1.00000 27 hdd 1.81898 osd.27 down 0 1.00000 30 hdd 1.81898 osd.30 down 0 1.00000 -1 47.21199 root default -40 23.59000 rack mainehall -3 23.59000 host ceph-osd1 0 hdd 1.81898 osd.0 up 1.00000 1.00000 1 hdd 1.81898 osd.1 up 1.00000 1.00000 2 hdd 1.81898 osd.2 up 1.00000 1.00000 4 hdd 1.81898 osd.4 up 1.00000 1.00000 5 hdd 1.81898 osd.5 up 0.90002 1.00000 6 hdd 1.81898 osd.6 up 1.00000 1.00000 7 hdd 1.81898 osd.7 up 1.00000 1.00000 8 hdd 1.81898 osd.8 up 1.00000 1.00000 9 hdd 1.81898 osd.9 up 1.00000 1.00000 10 hdd 1.81898 osd.10 up 0.95001 1.00000 33 hdd 1.76099 osd.33 up 1.00000 1.00000 38 hdd 3.63899 osd.38 up 1.00000 1.00000 -42 23.62199 rack rangleyhall -5 23.62199 host ceph-osd2 11 hdd 1.81898 osd.11 up 1.00000 1.00000 12 hdd 1.81898 osd.12 up 0.90002 1.00000 13 hdd 1.81898 osd.13 up 1.00000 1.00000 14 hdd 1.81898 osd.14 up 1.00000 1.00000 15 hdd 1.81898 osd.15 up 1.00000 1.00000 16 hdd 1.81898 osd.16 up 1.00000 1.00000 17 hdd 1.81898 osd.17 up 1.00000 1.00000 18 hdd 1.81898 osd.18 up 0.90002 1.00000 19 hdd 1.81898 osd.19 up 1.00000 1.00000 20 hdd 1.81898 osd.20 up 1.00000 1.00000 32 hdd 1.79300 osd.32 up 0.95001 1.00000 39 hdd 3.63899 osd.39 up 1.00000 1.00000 But it keeps changing to this: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -62 14.55199 root osd3-shelf2 -60 14.55199 chassis ceph-osd3-shelf2 3 hdd 1.81898 osd.3 down 1.00000 1.00000 40 hdd 1.81898 osd.40 down 1.00000 1.00000 41 hdd 1.81898 osd.41 down 1.00000 1.00000 42 hdd 1.81898 osd.42 down 1.00000 1.00000 43 hdd 1.81898 osd.43 down 1.00000 1.00000 44 hdd 1.81898 osd.44 down 1.00000 1.00000 45 hdd 1.81898 osd.45 down 1.00000 1.00000 46 hdd 1.81898 osd.46 down 1.00000 1.00000 -58 1.71599 root osd3-internal -54 1.71599 chassis ceph-osd3-internal 34 hdd 0.42899 osd.34 down 1.00000 1.00000 35 hdd 0.42899 osd.35 down 1.00000 1.00000 36 hdd 0.42899 osd.36 down 1.00000 1.00000 37 hdd 0.42899 osd.37 down 1.00000 1.00000 -50 14.55199 root osd3-shelf1 -56 14.55199 chassis ceph-osd3-shelf1 21 hdd 1.81898 osd.21 down 1.00000 1.00000 22 hdd 1.81898 osd.22 down 1.00000 1.00000 23 hdd 1.81898 osd.23 down 1.00000 1.00000 24 hdd 1.81898 osd.24 down 1.00000 1.00000 25 hdd 1.81898 osd.25 down 1.00000 1.00000 -7 5.45695 host ceph-osd3 26 hdd 1.81898 osd.26 down 0 1.00000 27 hdd 1.81898 osd.27 down 0 1.00000 30 hdd 1.81898 osd.30 down 0 1.00000 28 hdd 1.81898 osd.28 down 1.00000 1.00000 29 hdd 1.81898 osd.29 down 1.00000 1.00000 31 hdd 1.81898 osd.31 down 1.00000 1.00000 -1 47.21199 root default -40 23.59000 rack mainehall -3 23.59000 host ceph-osd1 0 hdd 1.81898 osd.0 up 1.00000 1.00000 1 hdd 1.81898 osd.1 up 1.00000 1.00000 2 hdd 1.81898 osd.2 up 1.00000 1.00000 4 hdd 1.81898 osd.4 up 1.00000 1.00000 5 hdd 1.81898 osd.5 up 0.90002 1.00000 6 hdd 1.81898 osd.6 up 1.00000 1.00000 7 hdd 1.81898 osd.7 up 1.00000 1.00000 8 hdd 1.81898 osd.8 up 1.00000 1.00000 9 hdd 1.81898 osd.9 up 1.00000 1.00000 10 hdd 1.81898 osd.10 up 0.95001 1.00000 33 hdd 1.76099 osd.33 up 1.00000 1.00000 38 hdd 3.63899 osd.38 up 1.00000 1.00000 -42 23.62199 rack rangleyhall -5 23.62199 host ceph-osd2 11 hdd 1.81898 osd.11 up 1.00000 1.00000 12 hdd 1.81898 osd.12 up 0.90002 1.00000 13 hdd 1.81898 osd.13 up 1.00000 1.00000 14 hdd 1.81898 osd.14 up 1.00000 1.00000 15 hdd 1.81898 osd.15 up 1.00000 1.00000 16 hdd 1.81898 osd.16 up 1.00000 1.00000 17 hdd 1.81898 osd.17 up 1.00000 1.00000 18 hdd 1.81898 osd.18 up 0.90002 1.00000 19 hdd 1.81898 osd.19 up 1.00000 1.00000 20 hdd 1.81898 osd.20 up 1.00000 1.00000 32 hdd 1.79300 osd.32 up 0.95001 1.00000 39 hdd 3.63899 osd.39 up 1.00000 1.00000 Robert Pelletier, IT and Security Specialist Eastern Maine Community College (207) 974-4782 | 354 Hogan Rd., Bangor, ME 04401
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
