Something very curious is that I was adjusting the configuration for osd memory target via ceph-ansible.... and had at one point set 2147483648 which is around 2GB
Currently It's set to 1610612736, but strangely in the config file it wrote 1963336226. But a very strange number shows up in the active sections of the pg's that's the same number roughly as 2147483648..... This seems very odd, and maybe the value got lodged somewhere it doesn't belong which is causing an issue. https://gist.github.com/arodd/c95355a7b55f3e4a94f21bc5e801943d#file-8-c-pg-query-L18 On Wed, Jul 3, 2019 at 1:08 PM Austin Workman <soilfla...@gmail.com> wrote: > So several events unfolded that may have led to this situation. Some of > them in hindsight were probably not the smartest decision around adjusting > the ec pool and restarting the OSD's several times during these migrations. > > > 1. Added a new 6th OSD with ceph-ansible > 1. Hung during restart of OSD's because they were set to noup and > one of the original OSD's wouldn't come back online because of the noup. > Manually unset noup and all 6 OSD's went up/in. > 2. Objects showing in degraded/misplaced > 3. Strange behavior restarting one OSD at a time and waiting for it to > stabilize, depending on which was the last OSD restarted, different > resulting backfill or move operations were taking place. > 4. Adjusted recovery/backfill sleep/concurrent moves to speed up > re-location. > 5. Decided that if all the data was going to move, I should adjust my > jerasure ec profile from k=4, m=1 -> k=5, m=1 with force(is this even > recommended vs. just creating new pools???) > 1. Initially it unset crush-device-class=hdd to be blank > 2. Re-set crush-device-class > 3. Couldn't determine if this had any effect on the move operations. > 4. Changed back to k=4 > 6. Let some of the backfill work through, ran into toofull situations > even though OSD's had plenty of space. > 1. Decided to add PG's to the EC pool 64->150 > 7. Restarted one OSD at a time again, waiting for them to be healthy > before moving on.(probably should have been setting noout) > 8. Eventually one of the old OSD's refused to start due to a thread > abort relating to stripe size(see below). > 9. Tried restarting other OSD's they all came back online fine. > 10. Some time passes and then the new OSD crashes, and won't start > back up with the same stripe size abort. > 1. Now 2 OSD's are down, and won't start back up due to that same > condition, and data is no longer available. > 2. 149 PG's showing as incomplete due to the min size 5(which > shouldn't it be 1 from the original EC/new EC profile settings?) > 3. 1 pg as down > 4. 21 unknown > 5. Some of the PG's were still "new pg's" from increasing the PG > size of the pool. > > So yeah, somewhat of a cluster of changing too many things at once here, > but I didn't realize the things I was doing would potentially have this > result. > > The two OSD's that won't start should still have all of the data on them, > it seems like they are having issues with at least one of the PG's in > particular from the EC pool that was adjusted, but presumably the rest of > the data should be fine, and hopefully there is a way to get them to start > up again. I saw a similar issue posted in the list a few years ago but > there was never any follow up from the user having the issue. > > https://gist.github.com/arodd/c95355a7b55f3e4a94f21bc5e801943d > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
