Hello List,
after some crash of a box, the journal vanished. Creating a new one
with --mkjournal results in the osd beeing unable to start.
Does anyone want to dissect this any further or should I just trash
the osd and recreate it?
Thx in advance
Benedikt
2015-12-01 07:46:31.505255 7fadb7f1e900 0 ceph version 0.94.5
(9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 5486
2015-12-01 07:46:31.628585 7fadb7f1e900 0
filestore(/var/lib/ceph/osd/ceph-328) backend xfs (magic 0x58465342)
2015-12-01 07:46:31.662972 7fadb7f1e900 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_features:
FIEMAP ioctl is supported and appears to work
2015-12-01 07:46:31.662984 7fadb7f1e900 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_features:
FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-12-01 07:46:31.674999 7fadb7f1e900 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_features:
syncfs(2) syscall fully supported (by glibc and kernel)
2015-12-01 07:46:31.675071 7fadb7f1e900 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_feature:
extsize is supported and kernel 3.19.0-33-generic >= 3.5
2015-12-01 07:46:31.806490 7fadb7f1e900 0
filestore(/var/lib/ceph/osd/ceph-328) mount: enabling WRITEAHEAD
journal mode: checkpoint is not enabled
2015-12-01 07:46:35.598698 7fadb7f1e900 1 journal _open
/var/lib/ceph/osd/ceph-328/journal fd 19: 9663676416 bytes, block size
4096 bytes, directio = 1, aio = 1
2015-12-01 07:46:35.600956 7fadb7f1e900 1 journal _open
/var/lib/ceph/osd/ceph-328/journal fd 19: 9663676416 bytes, block size
4096 bytes, directio = 1, aio = 1
2015-12-01 07:46:35.619860 7fadb7f1e900 0 <cls>
cls/hello/cls_hello.cc:271: loading cls_hello
2015-12-01 07:46:35.682532 7fadb7f1e900 -1 osd/OSD.h: In function
'OSDMapRef OSDService::get_map(epoch_t)' thread 7fadb7f1e900 time
2015-12-01 07:46:35.681204
osd/OSD.h: 716: FAILED assert(ret)
ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x8b) [0xbc60eb]
2: (OSDService::get_map(unsigned int)+0x3f) [0x70ad5f]
3: (OSD::init()+0x6ad) [0x6c5e0d]
4: (main()+0x2860) [0x6527e0]
5: (__libc_start_main()+0xf5) [0x7fadb505bec5]
6: /usr/bin/ceph-osd() [0x66b887]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- begin dump of recent events ---
-62> 2015-12-01 07:46:31.503728 7fadb7f1e900 5 asok(0x5402000)
register_command perfcounters_dump hook 0x53a2050
-61> 2015-12-01 07:46:31.503759 7fadb7f1e900 5 asok(0x5402000)
register_command 1 hook 0x53a2050
-60> 2015-12-01 07:46:31.503764 7fadb7f1e900 5 asok(0x5402000)
register_command perf dump hook 0x53a2050
-59> 2015-12-01 07:46:31.503768 7fadb7f1e900 5 asok(0x5402000)
register_command perfcounters_schema hook 0x53a2050
-58> 2015-12-01 07:46:31.503772 7fadb7f1e900 5 asok(0x5402000)
register_command 2 hook 0x53a2050
-57> 2015-12-01 07:46:31.503775 7fadb7f1e900 5 asok(0x5402000)
register_command perf schema hook 0x53a2050
-56> 2015-12-01 07:46:31.503786 7fadb7f1e900 5 asok(0x5402000)
register_command perf reset hook 0x53a2050
-55> 2015-12-01 07:46:31.503790 7fadb7f1e900 5 asok(0x5402000)
register_command config show hook 0x53a2050
-54> 2015-12-01 07:46:31.503792 7fadb7f1e900 5 asok(0x5402000)
register_command config set hook 0x53a2050
-53> 2015-12-01 07:46:31.503797 7fadb7f1e900 5 asok(0x5402000)
register_command config get hook 0x53a2050
-52> 2015-12-01 07:46:31.503799 7fadb7f1e900 5 asok(0x5402000)
register_command config diff hook 0x53a2050
-51> 2015-12-01 07:46:31.503802 7fadb7f1e900 5 asok(0x5402000)
register_command log flush hook 0x53a2050
-50> 2015-12-01 07:46:31.503804 7fadb7f1e900 5 asok(0x5402000)
register_command log dump hook 0x53a2050
-49> 2015-12-01 07:46:31.503807 7fadb7f1e900 5 asok(0x5402000)
register_command log reopen hook 0x53a2050
-48> 2015-12-01 07:46:31.505255 7fadb7f1e900 0 ceph version 0.94.5
(9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 5486
-47> 2015-12-01 07:46:31.619430 7fadb7f1e900 1 -- 10.9.246.104:0/0
learned my addr 10.9.246.104:0/0
-46> 2015-12-01 07:46:31.619439 7fadb7f1e900 1
accepter.accepter.bind my_inst.addr is 10.9.246.104:6821/5486
need_addr=0
-45> 2015-12-01 07:46:31.619457 7fadb7f1e900 1
accepter.accepter.bind my_inst.addr is 0.0.0.0:6824/5486 need_addr=1
-44> 2015-12-01 07:46:31.619473 7fadb7f1e900 1
accepter.accepter.bind my_inst.addr is 0.0.0.0:6825/5486 need_addr=1
-43> 2015-12-01 07:46:31.619492 7fadb7f1e900 1 -- 10.9.246.104:0/0
learned my addr 10.9.246.104:0/0
-42> 2015-12-01 07:46:31.619496 7fadb7f1e900 1
accepter.accepter.bind my_inst.addr is 10.9.246.104:6827/5486
need_addr=0
-41> 2015-12-01 07:46:31.620890 7fadb7f1e900 5 asok(0x5402000)
init /var/run/ceph/ceph-osd.328.asok
-40> 2015-12-01 07:46:31.620901 7fadb7f1e900 5 asok(0x5402000)
bind_and_listen /var/run/ceph/ceph-osd.328.asok
-39> 2015-12-01 07:46:31.620931 7fadb7f1e900 5 asok(0x5402000)
register_command 0 hook 0x539e0b0
-38> 2015-12-01 07:46:31.620937 7fadb7f1e900 5 asok(0x5402000)
register_command version hook 0x539e0b0
-37> 2015-12-01 07:46:31.620946 7fadb7f1e900 5 asok(0x5402000)
register_command git_version hook 0x539e0b0
-36> 2015-12-01 07:46:31.620954 7fadb7f1e900 5 asok(0x5402000)
register_command help hook 0x53a2140
-35> 2015-12-01 07:46:31.620962 7fadb7f1e900 5 asok(0x5402000)
register_command get_command_descriptions hook 0x53a2130
-34> 2015-12-01 07:46:31.621012 7fadb7f1e900 10 monclient(hunting):
build_initial_monmap
-33> 2015-12-01 07:46:31.621014 7fadb2024700 5 asok(0x5402000) entry start
-32> 2015-12-01 07:46:31.627732 7fadb7f1e900 5 adding auth protocol: cephx
-31> 2015-12-01 07:46:31.627740 7fadb7f1e900 5 adding auth protocol: cephx
-30> 2015-12-01 07:46:31.627832 7fadb7f1e900 5 asok(0x5402000)
register_command objecter_requests hook 0x53a2170
-29> 2015-12-01 07:46:31.627894 7fadb7f1e900 1 --
10.9.246.104:6821/5486 messenger.start
-28> 2015-12-01 07:46:31.627923 7fadb7f1e900 1 -- :/0 messenger.start
-27> 2015-12-01 07:46:31.627947 7fadb7f1e900 1 --
10.9.246.104:6827/5486 messenger.start
-26> 2015-12-01 07:46:31.627977 7fadb7f1e900 1 --
0.0.0.0:6825/5486 messenger.start
-25> 2015-12-01 07:46:31.628002 7fadb7f1e900 1 --
0.0.0.0:6824/5486 messenger.start
-24> 2015-12-01 07:46:31.628026 7fadb7f1e900 1 -- :/0 messenger.start
-23> 2015-12-01 07:46:31.628075 7fadb7f1e900 2 osd.328 0 mounting
/var/lib/ceph/osd/ceph-328 /var/lib/ceph/osd/ceph-328/journal
-22> 2015-12-01 07:46:31.628585 7fadb7f1e900 0
filestore(/var/lib/ceph/osd/ceph-328) backend xfs (magic 0x58465342)
-21> 2015-12-01 07:46:31.662972 7fadb7f1e900 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_features:
FIEMAP ioctl is supported and appears to work
-20> 2015-12-01 07:46:31.662984 7fadb7f1e900 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_features:
FIEMAP ioctl is disabled via 'filestore fiemap' config option
-19> 2015-12-01 07:46:31.674999 7fadb7f1e900 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_features:
syncfs(2) syscall fully supported (by glibc and kernel)
-18> 2015-12-01 07:46:31.675071 7fadb7f1e900 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_feature:
extsize is supported and kernel 3.19.0-33-generic >= 3.5
-17> 2015-12-01 07:46:31.806490 7fadb7f1e900 0
filestore(/var/lib/ceph/osd/ceph-328) mount: enabling WRITEAHEAD
journal mode: checkpoint is not enabled
-16> 2015-12-01 07:46:35.596667 7fadb7f1e900 2 journal open
/var/lib/ceph/osd/ceph-328/journal fsid
b054d6cb-8b2a-41aa-ac12-1169b42d036d fs_op_seq 2160370
-15> 2015-12-01 07:46:35.598698 7fadb7f1e900 1 journal _open
/var/lib/ceph/osd/ceph-328/journal fd 19: 9663676416 bytes, block size
4096 bytes, directio = 1, aio = 1
-14> 2015-12-01 07:46:35.598912 7fadb7f1e900 2 journal open
advancing committed_seq 0 to fs op_seq 2160370
-13> 2015-12-01 07:46:35.598947 7fadb7f1e900 2 journal No further
valid entries found, journal is most likely valid
-12> 2015-12-01 07:46:35.598954 7fadb7f1e900 2 journal No further
valid entries found, journal is most likely valid
-11> 2015-12-01 07:46:35.598956 7fadb7f1e900 3 journal
journal_replay: end of journal, done.
-10> 2015-12-01 07:46:35.600956 7fadb7f1e900 1 journal _open
/var/lib/ceph/osd/ceph-328/journal fd 19: 9663676416 bytes, block size
4096 bytes, directio = 1, aio = 1
-9> 2015-12-01 07:46:35.601366 7fadb7f1e900 2 osd.328 0 boot
-8> 2015-12-01 07:46:35.615322 7fadb7f1e900 1 <cls>
cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
-7> 2015-12-01 07:46:35.615507 7fadb7f1e900 1 <cls>
cls/statelog/cls_statelog.cc:306: Loaded log class!
-6> 2015-12-01 07:46:35.617645 7fadb7f1e900 1 <cls>
cls/rgw/cls_rgw.cc:3047: Loaded rgw class!
-5> 2015-12-01 07:46:35.619520 7fadb7f1e900 1 <cls>
cls/refcount/cls_refcount.cc:231: Loaded refcount class!
-4> 2015-12-01 07:46:35.619596 7fadb7f1e900 1 <cls>
cls/user/cls_user.cc:367: Loaded user class!
-3> 2015-12-01 07:46:35.619671 7fadb7f1e900 1 <cls>
cls/log/cls_log.cc:312: Loaded log class!
-2> 2015-12-01 07:46:35.619794 7fadb7f1e900 1 <cls>
cls/version/cls_version.cc:227: Loaded version class!
-1> 2015-12-01 07:46:35.619860 7fadb7f1e900 0 <cls>
cls/hello/cls_hello.cc:271: loading cls_hello
0> 2015-12-01 07:46:35.682532 7fadb7f1e900 -1 osd/OSD.h: In
function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7fadb7f1e900
time 2015-12-01 07:46:35.681204
osd/OSD.h: 716: FAILED assert(ret)
ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x8b) [0xbc60eb]
2: (OSDService::get_map(unsigned int)+0x3f) [0x70ad5f]
3: (OSD::init()+0x6ad) [0x6c5e0d]
4: (main()+0x2860) [0x6527e0]
5: (__libc_start_main()+0xf5) [0x7fadb505bec5]
6: /usr/bin/ceph-osd() [0x66b887]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
[...]
--- end dump of recent events ---
2015-12-01 07:46:35.684846 7fadb7f1e900 -1 *** Caught signal (Aborted) **
in thread 7fadb7f1e900
ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
1: /usr/bin/ceph-osd() [0xacd7ba]
2: (()+0x10340) [0x7fadb6bd1340]
3: (gsignal()+0x39) [0x7fadb5070cc9]
4: (abort()+0x148) [0x7fadb50740d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fadb597b535]
6: (()+0x5e6d6) [0x7fadb59796d6]
7: (()+0x5e703) [0x7fadb5979703]
8: (()+0x5e922) [0x7fadb5979922]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x278) [0xbc62d8]
10: (OSDService::get_map(unsigned int)+0x3f) [0x70ad5f]
11: (OSD::init()+0x6ad) [0x6c5e0d]
12: (main()+0x2860) [0x6527e0]
13: (__libc_start_main()+0xf5) [0x7fadb505bec5]
14: /usr/bin/ceph-osd() [0x66b887]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- begin dump of recent events ---
0> 2015-12-01 07:46:35.684846 7fadb7f1e900 -1 *** Caught signal
(Aborted) **
in thread 7fadb7f1e900
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
