On Wed, Jul 18, 2012 at 12:07 PM, Andrey Korolyov <[email protected]> wrote: > On Wed, Jul 18, 2012 at 10:30 PM, Gregory Farnum <[email protected]> wrote: >> On Wed, Jul 18, 2012 at 12:47 AM, Andrey Korolyov <[email protected]> wrote: >>> On Wed, Jul 18, 2012 at 11:18 AM, Gregory Farnum <[email protected]> wrote: >>>> On Tuesday, July 17, 2012 at 11:22 PM, Andrey Korolyov wrote: >>>>> On Wed, Jul 18, 2012 at 10:09 AM, Gregory Farnum <[email protected] >>>>> (mailto:[email protected])> wrote: >>>>> > Hrm. That shouldn't be possible if the OSD has been removed. How did >>>>> > you take it out? It sounds like maybe you just marked it in the OUT >>>>> > state (and turned it off quite quickly) without actually taking it out >>>>> > of the cluster? >>>>> > -Greg >>>>> >>>>> >>>>> >>>>> As I have did removal, it was definitely not like that - at first >>>>> place, I have marked osds(4 and 5 on same host) out, then rebuilt >>>>> crushmap and then kill osd processes. As I mentioned before, osd.4 >>>>> doest not exist in crushmap and therefore it shouldn`t be reported at >>>>> all(theoretically). >>>> >>>> Okay, that's what happened — marking an OSD out in the CRUSH map means all >>>> the data gets moved off it, but that doesn't remove it from all the places >>>> where it's registered in the monitor and in the map, for a couple reasons: >>>> 1) You might want to mark an OSD out before taking it down, to allow for >>>> more orderly data movement. >>>> 2) OSDs can get marked out automatically, but the system shouldn't be able >>>> to forget about them on its own. >>>> 3) You might want to remove an OSD from the CRUSH map in the process of >>>> placing it somewhere else (perhaps you moved the physical machine to a new >>>> location). >>>> etc. >>>> >>>> You want to run "ceph osd rm 4 5" and that should unregister both of them >>>> from everything[1]. :) >>>> -Greg >>>> [1]: Except for the full lists, which have a bug in the version of code >>>> you're running — remove the OSDs, then adjust the full ratios again, and >>>> all will be well. >>>> >>> >>> $ ceph osd rm 4 >>> osd.4 does not exist >>> $ ceph -s >>> health HEALTH_WARN 1 near full osd(s) >>> monmap e3: 3 mons at >>> {0=192.168.10.129:6789/0,1=192.168.10.128:6789/0,2=192.168.10.127:6789/0}, >>> election epoch 58, quorum 0,1,2 0,1,2 >>> osdmap e2198: 4 osds: 4 up, 4 in >>> pgmap v586056: 464 pgs: 464 active+clean; 66645 MB data, 231 GB >>> used, 95877 MB / 324 GB avail >>> mdsmap e207: 1/1/1 up {0=a=up:active} >>> >>> $ ceph health detail >>> HEALTH_WARN 1 near full osd(s) >>> osd.4 is near full at 89% >>> >>> $ ceph osd dump >>> .... >>> max_osd 4 >>> osd.0 up in weight 1 up_from 2183 up_thru 2187 down_at 2172 >>> last_clean_interval [2136,2171) 192.168.10.128:6800/4030 >>> 192.168.10.128:6801/4030 192.168.10.128:6802/4030 exists,up >>> 68b3deec-e80a-48b7-9c29-1b98f5de4f62 >>> osd.1 up in weight 1 up_from 2136 up_thru 2186 down_at 2135 >>> last_clean_interval [2115,2134) 192.168.10.129:6800/2980 >>> 192.168.10.129:6801/2980 192.168.10.129:6802/2980 exists,up >>> b2a26fe9-aaa8-445f-be1f-fa7d2a283b57 >>> osd.2 up in weight 1 up_from 2181 up_thru 2187 down_at 2172 >>> last_clean_interval [2136,2171) 192.168.10.128:6803/4128 >>> 192.168.10.128:6804/4128 192.168.10.128:6805/4128 exists,up >>> 378d367a-f7fb-4892-9ec9-db8ffdd2eb20 >>> osd.3 up in weight 1 up_from 2136 up_thru 2186 down_at 2135 >>> last_clean_interval [2115,2134) 192.168.10.129:6803/3069 >>> 192.168.10.129:6804/3069 192.168.10.129:6805/3069 exists,up >>> faf8eda8-55fc-4a0e-899f-47dbd32b81b8 >>> .... >> >> Hrm. How did you create your new crush map? All the normal avenues of >> removing an OSD from the map set a flag which the PGMap uses to delete >> its records (which would prevent it reappearing in the full list), and >> I can't see how setcrushmap would remove an OSD from the map (although >> there might be a code path I haven't found). > > Manually, by deleting osd4|5 entries and reweighing remaining nodes.
So you extracted the CRUSH map, edited it, and injected it using "ceph osd setrcrushmap"? -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
