On 16-09-13 11:13, Ronny Aasen wrote:
I suspect this must be a difficult question since there have been no replies on irc or mailinglist.

assuming it's impossible to get these osd's running again.

Is there a way to recover objects from the disks. ? they are mounted and data is readable. I have pg's down since they want to probe these osd's that do not want to start.

pg query claim it can continue if i mark the osd as lost. but i would prefer to not loose data. especially since the data is ok and readable on the nonfunctioning osd.

also let me know if there is other debug i can extract in order to troubleshoot the non starting osd's

kind regards
Ronny Aasen


I cannot help you with this, but you can try using http://ceph.com/community/incomplete-pgs-oh-my/ and http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-April/000238.html (found this mail thread googling for the objectool post). ymmv




On 12. sep. 2016 13:16, Ronny Aasen wrote:
after adding more osd's and having a big backfill running 2 of my osd's
keep on stopping.

We also recently upgraded from 0.94.7 to 0.94.9 but i do not know if
that is related.

the log say.

     0> 2016-09-12 10:31:08.288858 7f8749125880 -1 osd/PGLog.cc: In
function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t,
ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&,
PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&,
std::set<std::basic_string<char> >*)' thread 7f8749125880 time
2016-09-12 10:31:08.286337
osd/PGLog.cc: 984: FAILED assert(oi.version == i->first)

googeling led me to a bug that seems to be related to infernalis only.
dmesg does not show anything wrong with the hardware.

this is debian running hammer 0.94.9
and the osd is a software raid5 consisting of 5 3TB harddrives.
journal is a partition on ssd intel 3500

anyone have a clue to what can be wrong ?

kind regrads
Ronny Aasen





-- log debug_filestore=10 --
   -19> 2016-09-12 10:31:08.070947 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/1df4bfdd/rb.0.392c.238e1f29.0000002bd134/head '_' = 266
    -18> 2016-09-12 10:31:08.083111 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/deb5bfdd/rb.0.392c.238e1f29.0000002bc596/head '_' = 266
    -17> 2016-09-12 10:31:08.096718 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/9be5dfdd/rb.0.392c.238e1f29.0000002bc2bf/head '_' = 266
    -16> 2016-09-12 10:31:08.110048 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/cbf8ffdd/rb.0.392c.238e1f29.0000002b9d89/head '_' = 266
    -15> 2016-09-12 10:31:08.126263 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/e49d0fdd/rb.0.392c.238e1f29.0000002b078e/head '_' = 266
    -14> 2016-09-12 10:31:08.150199 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/e49d0fdd/rb.0.392c.238e1f29.0000002b078e/22 '_' = 259
    -13> 2016-09-12 10:31:08.173223 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/d0827fdd/rb.0.392c.238e1f29.0000002b0373/head '_' = 266
    -12> 2016-09-12 10:31:08.199192 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/d0827fdd/rb.0.392c.238e1f29.0000002b0373/22 '_' = 259
    -11> 2016-09-12 10:31:08.232712 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/bf4effdd/rb.0.392c.238e1f29.0000002ae882/head '_' = 266
    -10> 2016-09-12 10:31:08.265331 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/bf4effdd/rb.0.392c.238e1f29.0000002ae882/22 '_' = 259
     -9> 2016-09-12 10:31:08.265456 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) error opening file
/var/lib/ceph/osd/ceph-8/current/1.fdd_head/DIR_D/DIR_D/DIR_F/DIR_0/DIR_2/rb.0.392c.238e1f29.000000b381ae__head_DB220FDD__1
with flags=2: (2) No such file or directory
     -8> 2016-09-12 10:31:08.265475 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/db220fdd/rb.0.392c.238e1f29.000000b381ae/head '_' = -2
     -7> 2016-09-12 10:31:08.265535 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) error opening file
/var/lib/ceph/osd/ceph-8/current/1.fdd_head/DIR_D/DIR_D/DIR_F/DIR_0/DIR_2/rb.0.392c.238e1f29.000000b381ae__21_DB220FDD__1
with flags=2: (2) No such file or directory
     -6> 2016-09-12 10:31:08.265546 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/db220fdd/rb.0.392c.238e1f29.000000b381ae/21 '_' = -2
     -5> 2016-09-12 10:31:08.265609 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) error opening file
/var/lib/ceph/osd/ceph-8/current/1.fdd_head/DIR_D/DIR_D/DIR_F/DIR_0/DIR_2/rb.0.392c.238e1f29.000000cf4057__head_12020FDD__1
with flags=2: (2) No such file or directory
     -4> 2016-09-12 10:31:08.265628 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/12020fdd/rb.0.392c.238e1f29.000000cf4057/head '_' = -2
     -3> 2016-09-12 10:31:08.265688 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) error opening file
/var/lib/ceph/osd/ceph-8/current/1.fdd_head/DIR_D/DIR_D/DIR_F/DIR_0/DIR_2/rb.0.392c.238e1f29.000000cf4057__21_12020FDD__1
with flags=2: (2) No such file or directory
     -2> 2016-09-12 10:31:08.265700 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/12020fdd/rb.0.392c.238e1f29.000000cf4057/21 '_' = -2
     -1> 2016-09-12 10:31:08.286313 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/882e0fdd/rb.0.392c.238e1f29.0000003c9802/21 '_' = 251
      0> 2016-09-12 10:31:08.288858 7f8749125880 -1 osd/PGLog.cc: In
function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t,
ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&,
PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&,
std::set<std::basic_string<char> >*)' thread 7f8749125880 time
2016-09-12 10:31:08.286337
osd/PGLog.cc: 984: FAILED assert(oi.version == i->first)

  ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90)
  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x76) [0xc0f196]
  2: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t,
pg_info_t const&, std::map<eversion_t, hobject_t, std::less<eversion_t>,
std::allocator<std::pair<eversion_t const, hobject_t> > >&,
PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream<char,
std::char_traits<char>, std::allocator<char> >&, std::set<std::string,
std::less<std::string>, std::allocator<std::string> >*)+0x11ab) [0x76f9ab] 3: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x1e2) [0x7f6c72]
  4: (OSD::load_pgs()+0xac0) [0x6abd00]
  5: (OSD::init()+0x14da) [0x6af54a]
  6: (main()+0x2848) [0x6339f8]
  7: (__libc_start_main()+0xf5) [0x7f8746443b45]
  8: /usr/bin/ceph-osd() [0x64d687]
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

--- logging levels ---
    0/ 5 none
    0/ 1 lockdep
    0/ 1 context
    1/ 1 crush
    1/ 5 mds
    1/ 5 mds_balancer
    1/ 5 mds_locker
    1/ 5 mds_log
    1/ 5 mds_log_expire
    1/ 5 mds_migrator
    0/ 1 buffer
    0/ 1 timer
    0/ 1 filer
    0/ 1 striper
    0/ 1 objecter
    0/ 5 rados
    0/ 5 rbd
    0/ 5 rbd_replay
    0/ 5 journaler
    0/ 5 objectcacher
    0/ 5 client
    0/ 5 osd
    0/ 5 optracker
    0/ 5 objclass
   10/10 filestore
    1/ 3 keyvaluestore
    1/ 3 journal
    0/ 5 ms
    1/ 5 mon
    0/10 monc
    1/ 5 paxos
    0/ 5 tp
    1/ 5 auth
    1/ 5 crypto
    1/ 1 finisher
    1/ 5 heartbeatmap
    1/ 5 perfcounter
    1/ 5 rgw
    1/10 civetweb
    1/ 5 javaclient
    1/ 5 asok
    1/ 1 throttle
    0/ 0 refs
    1/ 5 xio
   -2/-2 (syslog threshold)
   -1/-1 (stderr threshold)
   max_recent     10000
   max_new         1000
   log_file /var/log/ceph/ceph-osd.8.log
--- end dump of recent events ---
2016-09-12 10:31:08.376098 7f8749125880 -1 *** Caught signal (Aborted) **
  in thread 7f8749125880

  ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90)
  1: /usr/bin/ceph-osd() [0xb0c4d3]
  2: (()+0xf8d0) [0x7f8747fb68d0]
  3: (gsignal()+0x37) [0x7f8746457067]
  4: (abort()+0x148) [0x7f8746458448]
  5: (__gnu_cxx::__verbose_terminate_handler()+0x15d) [0x7f8746d44b3d]
  6: (()+0x5ebb6) [0x7f8746d42bb6]
  7: (()+0x5ec01) [0x7f8746d42c01]
  8: (()+0x5ee19) [0x7f8746d42e19]
  9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x247) [0xc0f367]
  10: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t,
pg_info_t const&, std::map<eversion_t, hobject_t, std::less<eversion_t>,
std::allocator<std::pair<eversion_t const, hobject_t> > >&,
PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream<char,
std::char_traits<char>, std::allocator<char> >&, std::set<std::string,
std::less<std::string>, std::allocator<std::string> >*)+0x11ab) [0x76f9ab] 11: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x1e2) [0x7f6c72]
  12: (OSD::load_pgs()+0xac0) [0x6abd00]
  13: (OSD::init()+0x14da) [0x6af54a]
  14: (main()+0x2848) [0x6339f8]
  15: (__libc_start_main()+0xf5) [0x7f8746443b45]
  16: /usr/bin/ceph-osd() [0x64d687]
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

--- begin dump of recent events ---
      0> 2016-09-12 10:31:08.376098 7f8749125880 -1 *** Caught signal
(Aborted) **
  in thread 7f8749125880

  ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90)
  1: /usr/bin/ceph-osd() [0xb0c4d3]
  2: (()+0xf8d0) [0x7f8747fb68d0]
  3: (gsignal()+0x37) [0x7f8746457067]
  4: (abort()+0x148) [0x7f8746458448]
  5: (__gnu_cxx::__verbose_terminate_handler()+0x15d) [0x7f8746d44b3d]
  6: (()+0x5ebb6) [0x7f8746d42bb6]
  7: (()+0x5ec01) [0x7f8746d42c01]
  8: (()+0x5ee19) [0x7f8746d42e19]
  9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x247) [0xc0f367]
  10: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t,
pg_info_t const&, std::map<eversion_t, hobject_t, std::less<eversion_t>,
std::allocator<std::pair<eversion_t const, hobject_t> > >&,
PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream<char,
std::char_traits<char>, std::allocator<char> >&, std::set<std::string,
std::less<std::string>, std::allocator<std::string> >*)+0x11ab) [0x76f9ab] 11: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x1e2) [0x7f6c72]
  12: (OSD::load_pgs()+0xac0) [0x6abd00]
  13: (OSD::init()+0x14da) [0x6af54a]
  14: (main()+0x2848) [0x6339f8]
  15: (__libc_start_main()+0xf5) [0x7f8746443b45]
  16: /usr/bin/ceph-osd() [0x64d687]
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

--- logging levels ---
    0/ 5 none
    0/ 1 lockdep
    0/ 1 context
    1/ 1 crush
    1/ 5 mds
    1/ 5 mds_balancer
    1/ 5 mds_locker
    1/ 5 mds_log
    1/ 5 mds_log_expire
    1/ 5 mds_migrator
    0/ 1 buffer
    0/ 1 timer
    0/ 1 filer
    0/ 1 striper
    0/ 1 objecter
    0/ 5 rados
    0/ 5 rbd
    0/ 5 rbd_replay
    0/ 5 journaler
    0/ 5 objectcacher
    0/ 5 client
    0/ 5 osd
    0/ 5 optracker
    0/ 5 objclass
   10/10 filestore
    1/ 3 keyvaluestore
    1/ 3 journal
    0/ 5 ms
    1/ 5 mon
    0/10 monc
    1/ 5 paxos
    0/ 5 tp
    1/ 5 auth
    1/ 5 crypto
    1/ 1 finisher
    1/ 5 heartbeatmap
    1/ 5 perfcounter
    1/ 5 rgw
    1/10 civetweb
    1/ 5 javaclient
    1/ 5 asok
    1/ 1 throttle
    0/ 0 refs
    1/ 5 xio
   -2/-2 (syslog threshold)
   -1/-1 (stderr threshold)
   max_recent     10000
   max_new         1000
   log_file /var/log/ceph/ceph-osd.8.log
--- end dump of recent events ---
root@ceph-osd5:~#
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to