Hello,I had some faulty power cables on some OSD's in one server which caused
lots of IO issues/disks appearing/disappearing.This has been corrected now, 2
of the 10 OSD's are working, however 8 are failing to start due to what looks
to be a corrupt DB.When running a ceph-bluestore-tool fsck I get the following
output:rocksdb: [db/db_impl_open.cc:516] db.wal/002221.log: dropping 1302
bytes; Corruption: missing start of fragmented
record(2)2020-12-22T16:21:52.715+0100 7f7b6a1500c0 4 rocksdb:
[db/db_impl.cc:389] Shutdown: canceling all background
work2020-12-22T16:21:52.715+0100 7f7b6a1500c0 4 rocksdb: [db/db_impl.cc:563]
Shutdown complete2020-12-22T16:21:52.715+0100 7f7b6a1500c0 -1 rocksdb:
Corruption: missing start of fragmented record(2)2020-12-22T16:21:52.715+0100
7f7b6a1500c0 -1
bluestore(/var/lib/ceph/b1db6b36-0c4c-4bce-9cda-18834be0632d/osd.28) opendb
erroring opening db:Trying to start the OSD leads to:ceph_abort_msg("Bad table
magic number: expected 9863518390377041911, found 9372993859750765257 in
db/002442.sst")It looks like the last write to these OSD's never fully
completed, sadly as I was adding this new node to move from OSD to Host
redundancy (EC Pool) I have 20% down PG's currently, is there anything I can do
to remove the last entry in the DB or somehow clean up the rocksDB to get these
OSD's atleast started? Understand may end up with some corrupted files.Thanks
Sent via MXlogin
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
