On Wed, May 17, 2017 at 10:51 AM Steve Anthony <[email protected]> wrote:
> Hello, > > After starting a backup (create snap, export and import into a second > cluster - one RBD image still exporting/importing as of this message) > the other day while recovery operations on the primary cluster were > ongoing I noticed an OSD (osd.126) start to crash; I reweighted it to 0 > to prepare to remove it. Shortly thereafter I noticed the problem seemed > to move to another OSD (osd.223). After looking at the logs, I noticed > they appeared to have the same problem. I'm running Ceph version 9.2.1 > (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd) on Debian 8. > > Log for osd.126 from start to crash: https://pastebin.com/y4fn94xe > > Log for osd.223 from start to crash: https://pastebin.com/AE4CYvSA > > > May 15 10:39:55 ceph13 ceph-osd[21506]: -9308> 2017-05-15 > 10:39:51.561342 7f225c385900 -1 osd.126 616621 log_to_monitors > {default=true} > May 15 10:39:55 ceph13 ceph-osd[21506]: 2017-05-15 10:39:55.328897 > 7f2236be3700 -1 osd/ReplicatedPG.cc: In function 'virtual void > ReplicatedPG::on_local_recover(const hobject_t&, const > object_stat_sum_t&, const ObjectRecoveryInfo&, ObjectContextRef, > ObjectStore::Transaction*)' thread 7f2236be3700 time 2017-05-15 > 10:39:55.322306 > May 15 10:39:55 ceph13 ceph-osd[21506]: osd/ReplicatedPG.cc: 192: FAILED > assert(recovery_info.oi.snaps.size()) > > May 15 16:45:25 ceph19 ceph-osd[30527]: 2017-05-15 16:45:25.343391 > 7ff40f41e900 -1 osd.223 619808 log_to_monitors {default=true} > May 15 16:45:30 ceph19 ceph-osd[30527]: osd/ReplicatedPG.cc: In function > 'virtual void ReplicatedPG::on_local_recover(const hobject_t&, const > object_stat_sum_t&, const ObjectRecoveryInfo&, ObjectContextRef, > ObjectStore::Transaction*)' thread 7ff3eab63700 time 2017-05-15 > 16:45:30.799839 > May 15 16:45:30 ceph19 ceph-osd[30527]: osd/ReplicatedPG.cc: 192: FAILED > assert(recovery_info.oi.snaps.size()) > > > I did some searching and thought it might be related to > http://tracker.ceph.com/issues/13837 aka > https://bugzilla.redhat.com/show_bug.cgi?id=1351320 so I disabled > scrubbing and deep-scrubbing, and set osd_pg_max_concurrent_snap_trims > to 0 for all OSDs. No luck. I had changed the systemd service file to > automatically restart osd.223 while recovery was happening, but it > appears to have stalled; I suppose it's needed up for the remaining > objects. > Yeah, these aren't really related that I can see — though I haven't spent much time in this code that I can recall. The OSD is receiving a "push" as part of log recovery and finds that the object it's receiving is a snapshot object without having any information about the snap IDs that exist, which is weird. I don't know of any way a client could break it either, but maybe David or Jason know something more. -Greg > > I didn't see anything else online, so I thought I see if anyone has seen > this before or has any other ideas. Thanks for taking the time. > > -Steve > > > -- > Steve Anthony > LTS HPC Senior Analyst > Lehigh University > [email protected] > > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
