Noah, Is this fresh installation or after upgrade ? It seems related to omap (leveldb) stuff.
Thanks & Regards Somnath -----Original Message----- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Noah Mehl Sent: Sunday, March 22, 2015 9:34 AM To: ceph-users@lists.ceph.com Subject: [ceph-users] Can't Start OSD I have an OSD that’s failing to start. I can’t make heads or tails of the error (pasted below). Thanks! ~Noah 2015-03-22 16:32:39.265116 7f4da7fa0780 0 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7), process ceph-osd, pid 13483 2015-03-22 16:32:39.269499 7f4da7fa0780 1 filestore(/var/lib/ceph/osd/ceph-1) mount detected xfs 2015-03-22 16:32:39.269509 7f4da7fa0780 1 filestore(/var/lib/ceph/osd/ceph-1) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2015-03-22 16:32:39.450031 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is supported and appears to work 2015-03-22 16:32:39.450069 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-03-22 16:32:39.450743 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount did NOT detect btrfs 2015-03-22 16:32:39.499753 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount syncfs(2) syscall fully supported (by glibc and kernel) 2015-03-22 16:32:39.500078 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount found snaps <> 2015-03-22 16:32:40.765736 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: btrfs not detected 2015-03-22 16:32:40.777156 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 2551: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-03-22 16:32:40.777278 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 2551: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-03-22 16:32:40.778223 7f4da7fa0780 1 journal close /var/lib/ceph/osd/ceph-1/journal 2015-03-22 16:32:41.066655 7f4da7fa0780 1 filestore(/var/lib/ceph/osd/ceph-1) mount detected xfs 2015-03-22 16:32:41.150578 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is supported and appears to work 2015-03-22 16:32:41.150624 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-03-22 16:32:41.151359 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount did NOT detect btrfs 2015-03-22 16:32:41.225302 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount syncfs(2) syscall fully supported (by glibc and kernel) 2015-03-22 16:32:41.225498 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount found snaps <> 2015-03-22 16:32:42.375558 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: btrfs not detected 2015-03-22 16:32:42.382958 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 1429: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-03-22 16:32:42.383187 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 1481: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-03-22 16:32:43.076434 7f4da7fa0780 -1 osd/PG.cc: In function 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::bufferlist*)' thread 7f4da7fa0780 time 2015-03-22 16:32:43.075101 osd/PG.cc: 2270: FAILED assert(values.size() == 1) ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) 1: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::buffer::list*)+0x4d7) [0x70ebf7] 2: (OSD::load_pgs()+0x14ce) [0x694efe] 3: (OSD::init()+0x11be) [0x69cffe] 4: (main()+0x1d09) [0x5c3509] 5: (__libc_start_main()+0xed) [0x7f4da5bde76d] 6: /usr/bin/ceph-osd() [0x5c6e1d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -75> 2015-03-22 16:32:39.259280 7f4da7fa0780 5 asok(0x1aec1c0) register_command perfcounters_dump hook 0x1ae4010 -74> 2015-03-22 16:32:39.259373 7f4da7fa0780 5 asok(0x1aec1c0) register_command 1 hook 0x1ae4010 -73> 2015-03-22 16:32:39.259393 7f4da7fa0780 5 asok(0x1aec1c0) register_command perf dump hook 0x1ae4010 -72> 2015-03-22 16:32:39.259429 7f4da7fa0780 5 asok(0x1aec1c0) register_command perfcounters_schema hook 0x1ae4010 -71> 2015-03-22 16:32:39.259445 7f4da7fa0780 5 asok(0x1aec1c0) register_command 2 hook 0x1ae4010 -70> 2015-03-22 16:32:39.259453 7f4da7fa0780 5 asok(0x1aec1c0) register_command perf schema hook 0x1ae4010 -69> 2015-03-22 16:32:39.259467 7f4da7fa0780 5 asok(0x1aec1c0) register_command config show hook 0x1ae4010 -68> 2015-03-22 16:32:39.259481 7f4da7fa0780 5 asok(0x1aec1c0) register_command config set hook 0x1ae4010 -67> 2015-03-22 16:32:39.259495 7f4da7fa0780 5 asok(0x1aec1c0) register_command config get hook 0x1ae4010 -66> 2015-03-22 16:32:39.259505 7f4da7fa0780 5 asok(0x1aec1c0) register_command log flush hook 0x1ae4010 -65> 2015-03-22 16:32:39.259519 7f4da7fa0780 5 asok(0x1aec1c0) register_command log dump hook 0x1ae4010 -64> 2015-03-22 16:32:39.259536 7f4da7fa0780 5 asok(0x1aec1c0) register_command log reopen hook 0x1ae4010 -63> 2015-03-22 16:32:39.265116 7f4da7fa0780 0 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7), process ceph-osd, pid 13483 -62> 2015-03-22 16:32:39.266443 7f4da7fa0780 1 -- 192.168.41.42:0/0 learned my addr 192.168.41.42:0/0 -61> 2015-03-22 16:32:39.266462 7f4da7fa0780 1 accepter.accepter.bind my_inst.addr is 192.168.41.42:6803/13483 need_addr=0 -60> 2015-03-22 16:32:39.266500 7f4da7fa0780 1 -- 192.168.42.42:0/0 learned my addr 192.168.42.42:0/0 -59> 2015-03-22 16:32:39.266537 7f4da7fa0780 1 accepter.accepter.bind my_inst.addr is 192.168.42.42:6802/13483 need_addr=0 -58> 2015-03-22 16:32:39.266551 7f4da7fa0780 1 -- 192.168.42.42:0/0 learned my addr 192.168.42.42:0/0 -57> 2015-03-22 16:32:39.266560 7f4da7fa0780 1 accepter.accepter.bind my_inst.addr is 192.168.42.42:6803/13483 need_addr=0 -56> 2015-03-22 16:32:39.266580 7f4da7fa0780 1 -- 192.168.41.42:0/0 learned my addr 192.168.41.42:0/0 -55> 2015-03-22 16:32:39.266602 7f4da7fa0780 1 accepter.accepter.bind my_inst.addr is 192.168.41.42:6808/13483 need_addr=0 -54> 2015-03-22 16:32:39.269108 7f4da7fa0780 5 asok(0x1aec1c0) init /var/run/ceph/ceph-osd.1.asok -53> 2015-03-22 16:32:39.269138 7f4da7fa0780 5 asok(0x1aec1c0) bind_and_listen /var/run/ceph/ceph-osd.1.asok -52> 2015-03-22 16:32:39.269185 7f4da7fa0780 5 asok(0x1aec1c0) register_command 0 hook 0x1ae30b0 -51> 2015-03-22 16:32:39.269203 7f4da7fa0780 5 asok(0x1aec1c0) register_command version hook 0x1ae30b0 -50> 2015-03-22 16:32:39.269206 7f4da7fa0780 5 asok(0x1aec1c0) register_command git_version hook 0x1ae30b0 -49> 2015-03-22 16:32:39.269210 7f4da7fa0780 5 asok(0x1aec1c0) register_command help hook 0x1ae40d0 -48> 2015-03-22 16:32:39.269231 7f4da7fa0780 5 asok(0x1aec1c0) register_command get_command_descriptions hook 0x1ae40c0 -47> 2015-03-22 16:32:39.269273 7f4da3c28700 5 asok(0x1aec1c0) entry start -46> 2015-03-22 16:32:39.269499 7f4da7fa0780 1 filestore(/var/lib/ceph/osd/ceph-1) mount detected xfs -45> 2015-03-22 16:32:39.269509 7f4da7fa0780 1 filestore(/var/lib/ceph/osd/ceph-1) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs -44> 2015-03-22 16:32:39.450031 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is supported and appears to work -43> 2015-03-22 16:32:39.450069 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option -42> 2015-03-22 16:32:39.450743 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount did NOT detect btrfs -41> 2015-03-22 16:32:39.499753 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount syncfs(2) syscall fully supported (by glibc and kernel) -40> 2015-03-22 16:32:39.500078 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount found snaps <> -39> 2015-03-22 16:32:40.765736 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: btrfs not detected -38> 2015-03-22 16:32:40.777088 7f4da7fa0780 2 journal open /var/lib/ceph/osd/ceph-1/journal fsid e2ad61ec-c581-4159-8671-bab77c7d4e97 fs_op_seq 81852815 -37> 2015-03-22 16:32:40.777156 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 2551: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1 -36> 2015-03-22 16:32:40.777242 7f4da7fa0780 2 journal No further valid entries found, journal is most likely valid -35> 2015-03-22 16:32:40.777252 7f4da7fa0780 2 journal No further valid entries found, journal is most likely valid -34> 2015-03-22 16:32:40.777255 7f4da7fa0780 3 journal journal_replay: end of journal, done. -33> 2015-03-22 16:32:40.777278 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 2551: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1 -32> 2015-03-22 16:32:40.777874 7f4d9fc20700 1 FileStore::op_tp worker finish -31> 2015-03-22 16:32:40.777930 7f4da0421700 1 FileStore::op_tp worker finish -30> 2015-03-22 16:32:40.778223 7f4da7fa0780 1 journal close /var/lib/ceph/osd/ceph-1/journal -29> 2015-03-22 16:32:41.066043 7f4da7fa0780 10 monclient(hunting): build_initial_monmap -28> 2015-03-22 16:32:41.066137 7f4da7fa0780 5 adding auth protocol: cephx -27> 2015-03-22 16:32:41.066147 7f4da7fa0780 5 adding auth protocol: cephx -26> 2015-03-22 16:32:41.066384 7f4da7fa0780 1 -- 192.168.41.42:6803/13483 messenger.start -25> 2015-03-22 16:32:41.066418 7f4da7fa0780 1 -- :/0 messenger.start -24> 2015-03-22 16:32:41.066444 7f4da7fa0780 1 -- 192.168.41.42:6808/13483 messenger.start -23> 2015-03-22 16:32:41.066469 7f4da7fa0780 1 -- 192.168.42.42:6803/13483 messenger.start -22> 2015-03-22 16:32:41.066512 7f4da7fa0780 1 -- 192.168.42.42:6802/13483 messenger.start -21> 2015-03-22 16:32:41.066610 7f4da7fa0780 2 osd.1 0 mounting /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal -20> 2015-03-22 16:32:41.066655 7f4da7fa0780 1 filestore(/var/lib/ceph/osd/ceph-1) mount detected xfs -19> 2015-03-22 16:32:41.150578 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is supported and appears to work -18> 2015-03-22 16:32:41.150624 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option -17> 2015-03-22 16:32:41.151359 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount did NOT detect btrfs -16> 2015-03-22 16:32:41.225302 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount syncfs(2) syscall fully supported (by glibc and kernel) -15> 2015-03-22 16:32:41.225498 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount found snaps <> -14> 2015-03-22 16:32:42.375558 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: btrfs not detected -13> 2015-03-22 16:32:42.382825 7f4da7fa0780 2 journal open /var/lib/ceph/osd/ceph-1/journal fsid e2ad61ec-c581-4159-8671-bab77c7d4e97 fs_op_seq 81852815 -12> 2015-03-22 16:32:42.382958 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 1429: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1 -11> 2015-03-22 16:32:42.383091 7f4da7fa0780 2 journal No further valid entries found, journal is most likely valid -10> 2015-03-22 16:32:42.383108 7f4da7fa0780 2 journal No further valid entries found, journal is most likely valid -9> 2015-03-22 16:32:42.383111 7f4da7fa0780 3 journal journal_replay: end of journal, done. -8> 2015-03-22 16:32:42.383187 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 1481: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1 -7> 2015-03-22 16:32:42.383761 7f4da7fa0780 2 osd.1 0 boot -6> 2015-03-22 16:32:42.388322 7f4da7fa0780 1 <cls> cls/rgw/cls_rgw.cc:1596: Loaded rgw class! -5> 2015-03-22 16:32:42.389272 7f4da7fa0780 1 <cls> cls/log/cls_log.cc:313: Loaded log class! -4> 2015-03-22 16:32:42.392742 7f4da7fa0780 1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class! -3> 2015-03-22 16:32:42.393520 7f4da7fa0780 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class! -2> 2015-03-22 16:32:42.394181 7f4da7fa0780 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class! -1> 2015-03-22 16:32:42.394476 7f4da7fa0780 1 <cls> cls/version/cls_version.cc:227: Loaded version class! 0> 2015-03-22 16:32:43.076434 7f4da7fa0780 -1 osd/PG.cc: In function 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::bufferlist*)' thread 7f4da7fa0780 time 2015-03-22 16:32:43.075101 osd/PG.cc: 2270: FAILED assert(values.size() == 1) ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) 1: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::buffer::list*)+0x4d7) [0x70ebf7] 2: (OSD::load_pgs()+0x14ce) [0x694efe] 3: (OSD::init()+0x11be) [0x69cffe] 4: (main()+0x1d09) [0x5c3509] 5: (__libc_start_main()+0xed) [0x7f4da5bde76d] 6: /usr/bin/ceph-osd() [0x5c6e1d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.1.log --- end dump of recent events --- 2015-03-22 16:32:43.079587 7f4da7fa0780 -1 *** Caught signal (Aborted) ** in thread 7f4da7fa0780 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) 1: /usr/bin/ceph-osd() [0x8001ea] 2: (()+0xfcb0) [0x7f4da743acb0] 3: (gsignal()+0x35) [0x7f4da5bf3425] 4: (abort()+0x17b) [0x7f4da5bf6b8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4da654569d] 6: (()+0xb5846) [0x7f4da6543846] 7: (()+0xb5873) [0x7f4da6543873] 8: (()+0xb596e) [0x7f4da654396e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x8c5e7f] 10: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::buffer::list*)+0x4d7) [0x70ebf7] 11: (OSD::load_pgs()+0x14ce) [0x694efe] 12: (OSD::init()+0x11be) [0x69cffe] 13: (main()+0x1d09) [0x5c3509] 14: (__libc_start_main()+0xed) [0x7f4da5bde76d] 15: /usr/bin/ceph-osd() [0x5c6e1d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- 0> 2015-03-22 16:32:43.079587 7f4da7fa0780 -1 *** Caught signal (Aborted) ** in thread 7f4da7fa0780 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) 1: /usr/bin/ceph-osd() [0x8001ea] 2: (()+0xfcb0) [0x7f4da743acb0] 3: (gsignal()+0x35) [0x7f4da5bf3425] 4: (abort()+0x17b) [0x7f4da5bf6b8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4da654569d] 6: (()+0xb5846) [0x7f4da6543846] 7: (()+0xb5873) [0x7f4da6543873] 8: (()+0xb596e) [0x7f4da654396e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x8c5e7f] 10: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::buffer::list*)+0x4d7) [0x70ebf7] 11: (OSD::load_pgs()+0x14ce) [0x694efe] 12: (OSD::init()+0x11be) [0x69cffe] 13: (main()+0x1d09) [0x5c3509] 14: (__libc_start_main()+0xed) [0x7f4da5bde76d] 15: /usr/bin/ceph-osd() [0x5c6e1d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.1.log --- end dump of recent events --- _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
