I am trying to setup a Ceph cluster on 4 odroid-hc2 instances on top of
Ubuntu 18.04.
My ceph-mgr deamon keeps crashing on me.
Any advise on how to proceed?
Log on mgr node says something about ms_dispatch:
2019-05-20 15:34:43.070424 b6714230 0 set uid:gid to 64045:64045
(ceph:ceph)
2019-05-20 15:34:43.070455 b6714230 0 ceph version 12.2.11
(26dc3775efc7bb286a1d6d66faee0b
a30ea23eee) luminous (stable), process ceph-mgr, pid 1169
2019-05-20 15:34:43.070799 b6714230 0 pidfile_write: ignore empty
--pid-file
2019-05-20 15:34:43.101162 b6714230 1 mgr send_beacon standby
2019-05-20 15:34:43.124462 b06f8c30 -1 *** Caught signal (Segmentation
fault) **
in thread b06f8c30 thread_name:ms_dispatch
ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous
(stable)
1: (()+0x30133c) [0x77033c]
2: (()+0x25750) [0xb688a750]
3: (_ULarm_step()+0x55) [0xb6816ce6]
4: (()+0x255e8) [0xb6cd85e8]
5: (GetStackTrace(void**, int, int)+0x25) [0xb6cd8a3e]
6: (tcmalloc::PageHeap::GrowHeap(unsigned int)+0xb9) [0xb6ccd36a]
7: (tcmalloc::PageHeap::New(unsigned int)+0x79) [0xb6ccd5e6]
8: (tcmalloc::CentralFreeList::Populate()+0x71) [0xb6ccc5ce]
9: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**,
void**)+0x1b) [0xb6ccc76
0]
10: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x6d)
[0xb6ccc7de]
11: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, unsigned
int)+0x51) [0xb6c
cea56]
12: (malloc()+0x22d) [0xb6cd9a8e]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this
.
--- begin dump of recent events ---
-90> 2019-05-20 15:34:43.053293 b6714230 5 asok(0x55b5320)
register_command perfcounter
s_dump hook 0x554c088
-89> 2019-05-20 15:34:43.053322 b6714230 5 asok(0x55b5320)
register_command 1 hook 0x55
4c088
-88> 2019-05-20 15:34:43.053330 b6714230 5 asok(0x55b5320)
register_command perf dump h
ook 0x554c088
-87> 2019-05-20 15:34:43.053341 b6714230 5 asok(0x55b5320)
register_command perfcounter
s_schema hook 0x554c088
-86> 2019-05-20 15:34:43.053360 b6714230 5 asok(0x55b5320)
register_command perf histog
ram dump hook 0x554c088
-85> 2019-05-20 15:34:43.053374 b6714230 5 asok(0x55b5320)
register_command 2 hook 0x55
4c088
-84> 2019-05-20 15:34:43.053381 b6714230 5 asok(0x55b5320)
register_command perf schema
hook 0x554c088
-83> 2019-05-20 15:34:43.053389 b6714230 5 asok(0x55b5320)
register_command perf histog
ram schema hook 0x554c088
-82> 2019-05-20 15:34:43.053410 b6714230 5 asok(0x55b5320)
register_command perf reset
hook 0x554c088
-81> 2019-05-20 15:34:43.053418 b6714230 5 asok(0x55b5320)
register_command config show
hook 0x554c088
-80> 2019-05-20 15:34:43.053425 b6714230 5 asok(0x55b5320)
register_command config help
hook 0x554c088
-79> 2019-05-20 15:34:43.053436 b6714230 5 asok(0x55b5320)
register_command config set
hook 0x554c088
-78> 2019-05-20 15:34:43.053444 b6714230 5 asok(0x55b5320)
register_command config get
hook 0x554c088
-77> 2019-05-20 15:34:43.053459 b6714230 5 asok(0x55b5320)
register_command config diff
hook 0x554c088
-76> 2019-05-20 15:34:43.053467 b6714230 5 asok(0x55b5320)
register_command config diff
get hook 0x554c088
-75> 2019-05-20 15:34:43.053475 b6714230 5 asok(0x55b5320)
register_command log flush h
ook 0x554c088
-74> 2019-05-20 15:34:43.053482 b6714230 5 asok(0x55b5320)
register_command log dump ho
ok 0x554c088
-73> 2019-05-20 15:34:43.053490 b6714230 5 asok(0x55b5320)
register_command log reopen
hook 0x554c088
-72> 2019-05-20 15:34:43.053513 b6714230 5 asok(0x55b5320)
register_command dump_mempoo
ls hook 0x56e3504
-71> 2019-05-20 15:34:43.070424 b6714230 0 set uid:gid to 64045:64045
(ceph:ceph)
-70> 2019-05-20 15:34:43.070455 b6714230 0 ceph version 12.2.11
(26dc3775efc7bb286a1d6d
66faee0ba30ea23eee) luminous (stable), process ceph-mgr, pid 1169
-69> 2019-05-20 15:34:43.070799 b6714230 0 pidfile_write: ignore empty
--pid-file
-68> 2019-05-20 15:34:43.074441 b6714230 5 asok(0x55b5320) init
/var/run/ceph/ceph-mgr.
odroid-c.asok
-67> 2019-05-20 15:34:43.074473 b6714230 5 asok(0x55b5320)
bind_and_listen /var/run/cep
h/ceph-mgr.odroid-c.asok
-66> 2019-05-20 15:34:43.074615 b6714230 5 asok(0x55b5320)
register_command 0 hook 0x55
4c1d0
-65> 2019-05-20 15:34:43.074633 b6714230 5 asok(0x55b5320)
register_command version hoo
k 0x554c1d0
-64> 2019-05-20 15:34:43.074654 b6714230 5 asok(0x55b5320)
register_command git_version
hook 0x554c1d0
-63> 2019-05-20 15:34:43.074674 b6714230 5 asok(0x55b5320)
register_command help hook 0
x554c1d8
-62> 2019-05-20 15:34:43.074694 b6714230 5 asok(0x55b5320)
register_command get_command
_descriptions hook 0x554c1e0
-61> 2019-05-20 15:34:43.074785 b3effc30 5 asok(0x55b5320) entry start
-60> 2019-05-20 15:34:43.076464 b36fec30 2 Event(0x554e068 nevent=5000
time_id=1).set_o
wner idx=0 owner=3010456624
-59> 2019-05-20 15:34:43.076559 b2efdc30 2 Event(0x554e488 nevent=5000
time_id=1).set_o
wner idx=1 owner=3002063920
-58> 2019-05-20 15:34:43.076643 b26fcc30 2 Event(0x554e1c8 nevent=5000
time_id=1).set_o
wner idx=2 owner=2993671216
-57> 2019-05-20 15:34:43.077177 b6714230 1 Processor -- start
-56> 2019-05-20 15:34:43.077298 b6714230 1 -- - start start
-55> 2019-05-20 15:34:43.077315 b6714230 10 monclient:
build_initial_monmap
-54> 2019-05-20 15:34:43.077362 b6714230 10 monclient: init
-53> 2019-05-20 15:34:43.077380 b6714230 5 adding auth protocol: cephx
-52> 2019-05-20 15:34:43.077391 b6714230 10 monclient: auth_supported 2
method cephx
-51> 2019-05-20 15:34:43.077625 b6714230 2 auth: KeyRing::load: loaded
key file /var/li
b/ceph/mgr/ceph-odroid-c/keyring
-50> 2019-05-20 15:34:43.077761 b6714230 10 monclient: _reopen_session
rank -1
-49> 2019-05-20 15:34:43.077847 b6714230 10 monclient(hunting): picked
mon.noname-a con
0x5792d00 addr 192.168.130.131:6789/0
-48> 2019-05-20 15:34:43.077899 b6714230 1 -- - -->
192.168.130.131:6789/0 -- auth(prot
o 0 33 bytes epoch 0) v1 -- 0x5590680 con 0
-47> 2019-05-20 15:34:43.077985 b6714230 10 monclient(hunting):
_renew_subs
-46> 2019-05-20 15:34:43.080980 b2efdc30 1 --
192.168.130.132:0/2049423493 learned_addr
learned my addr 192.168.130.132:0/2049423493
-45> 2019-05-20 15:34:43.082020 b2efdc30 2 --
192.168.130.132:0/2049423493 >> 192.168.1
30.131:6789/0 conn(0x5792d00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0
l=0)._process_c
onnection got newly_acked_seq 0 vs out_seq 0
-44> 2019-05-20 15:34:43.084528 b2efdc30 5 --
192.168.130.132:0/2049423493 >> 192.168.1
30.131:6789/0 conn(0x5792d00 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=45 cs=1
l=1). rx mon.0 seq 1 0x55aa900 mon_map magic: 0 v1
-43> 2019-05-20 15:34:43.084615 b06f8c30 1 --
192.168.130.132:0/2049423493 <== mon.0 19
2.168.130.131:6789/0 1 ==== mon_map magic: 0 v1 ==== 196+0+0 (1694575244 0
0) 0x55aa900 con
0x5792d00
-42> 2019-05-20 15:34:43.084656 b06f8c30 10 monclient(hunting):
handle_monmap mon_map ma
gic: 0 v1
-41> 2019-05-20 15:34:43.084685 b06f8c30 10 monclient(hunting): got
monmap 1, mon.nonam
e-a is now rank -1
-40> 2019-05-20 15:34:43.084698 b06f8c30 10 monclient(hunting): dump:
epoch 1
fsid 75cb9a2d-673b-4a32-897a-05470a08ed58
last_changed 2019-05-20 15:02:53.998735
created 2019-05-20 15:02:53.998735
0: 192.168.130.131:6789/0 mon.odroid-b
-39> 2019-05-20 15:34:43.084956 b2efdc30 5 --
192.168.130.132:0/2049423493 >> 192.168.1
30.131:6789/0 conn(0x5792d00 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=45 cs=1
l=1). rx mon.0 seq 2 0x55a0540 auth_reply(proto 2 0 (0) Success) v1
-38> 2019-05-20 15:34:43.085011 b06f8c30 1 --
192.168.130.132:0/2049423493 <== mon.0 19
2.168.130.131:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ====
33+0+0 (4086221156 0
0) 0x55a0540 con 0x5792d00
-37> 2019-05-20 15:34:43.085053 b06f8c30 10 monclient(hunting): my
global_id is 24139
-36> 2019-05-20 15:34:43.085175 b06f8c30 1 --
192.168.130.132:0/2049423493 --> 192.168.
130.131:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x5590d00 con 0
-35> 2019-05-20 15:34:43.088488 b2efdc30 5 --
192.168.130.132:0/2049423493 >> 192.168.1
30.131:6789/0 conn(0x5792d00 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=45 cs=1
l=1). rx mon.0 seq 3 0x55a0700 auth_reply(proto 2 0 (0) Success) v1
-34> 2019-05-20 15:34:43.088712 b06f8c30 1 --
192.168.130.132:0/2049423493 <== mon.0 19
2.168.130.131:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ====
222+0+0 (1945430716 0
0) 0x55a0700 con 0x5792d00
-33> 2019-05-20 15:34:43.089295 b06f8c30 1 --
192.168.130.132:0/2049423493 --> 192.168.
130.131:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- 0x5590680 con 0
-32> 2019-05-20 15:34:43.097488 b2efdc30 5 --
192.168.130.132:0/2049423493 >> 192.168.1
30.131:6789/0 conn(0x5792d00 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=45 cs=1
l=1). rx mon.0 seq 4 0x55a08c0 auth_reply(proto 2 0 (0) Success) v1
-31> 2019-05-20 15:34:43.097643 b06f8c30 1 --
192.168.130.132:0/2049423493 <== mon.0 19
2.168.130.131:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ====
783+0+0 (327382700 0
0) 0x55a08c0 con 0x5792d00
-30> 2019-05-20 15:34:43.098725 b06f8c30 1 monclient: found mon.odroid-b
-29> 2019-05-20 15:34:43.098850 b06f8c30 10 monclient: _send_mon_message
to mon.odroid-b
at 192.168.130.131:6789/0
-28> 2019-05-20 15:34:43.098898 b06f8c30 1 --
192.168.130.132:0/2049423493 --> 192.168.
130.131:6789/0 -- mon_subscribe({mgrmap=0+,monmap=0+}) v2 -- 0x554eb00 con
0
-27> 2019-05-20 15:34:43.099042 b06f8c30 10 monclient:
_check_auth_rotating renewing rot
ating keys (they expired before 2019-05-20 15:34:13.099036)
-26> 2019-05-20 15:34:43.099183 b06f8c30 10 monclient: _send_mon_message
to mon.odroid-b
at 192.168.130.131:6789/0
-25> 2019-05-20 15:34:43.099271 b06f8c30 1 --
192.168.130.132:0/2049423493 --> 192.168.
130.131:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x5590d00 con 0
-24> 2019-05-20 15:34:43.099404 b6714230 5 monclient: authenticate
success, global_id 2
4139
-23> 2019-05-20 15:34:43.099543 b6714230 10 log_channel(cluster)
update_config to_monito
rs: true to_syslog: false syslog_facility: daemon prio: info to_graylog:
false graylog_host
: 127.0.0.1 graylog_port: 12201)
-22> 2019-05-20 15:34:43.099602 b6714230 10 log_channel(audit)
update_config to_monitors
: true to_syslog: false syslog_facility: local0 prio: info to_graylog:
false graylog_host:
127.0.0.1 graylog_port: 12201)
-21> 2019-05-20 15:34:43.099970 b6714230 5 asok(0x55b5320)
register_command objecter_re
quests hook 0x554c238
-20> 2019-05-20 15:34:43.100171 b6714230 10 monclient: _renew_subs
-19> 2019-05-20 15:34:43.100214 b6714230 10 monclient: _send_mon_message
to mon.odroid-b
at 192.168.130.131:6789/0
-18> 2019-05-20 15:34:43.100246 b6714230 1 --
192.168.130.132:0/2049423493 --> 192.168.
130.131:6789/0 -- mon_subscribe({osdmap=0}) v2 -- 0x554ec60 con 0
-17> 2019-05-20 15:34:43.100737 b6714230 5 asok(0x55b5320)
register_command mds_request
s hook 0xbefefe80
-16> 2019-05-20 15:34:43.100793 b6714230 5 asok(0x55b5320)
register_command mds_session
s hook 0xbefefe80
-15> 2019-05-20 15:34:43.100847 b6714230 5 asok(0x55b5320)
register_command dump_cache
hook 0xbefefe80
-14> 2019-05-20 15:34:43.100811 b2efdc30 5 --
192.168.130.132:0/2049423493 >> 192.168.1
30.131:6789/0 conn(0x5792d00 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=45 cs=1
l=1). rx mon.0 seq 5 0x558dc00 mgrmap(e 99) v1
-13> 2019-05-20 15:34:43.100915 b6714230 5 asok(0x55b5320)
register_command kick_stale_
sessions hook 0xbefefe80
-12> 2019-05-20 15:34:43.100977 b6714230 5 asok(0x55b5320)
register_command status hook
0xbefefe80
-11> 2019-05-20 15:34:43.100987 b06f8c30 1 --
192.168.130.132:0/2049423493 <== mon.0 19
2.168.130.131:6789/0 5 ==== mgrmap(e 99) v1 ==== 232+0+0 (4078310027 0 0)
0x558dc00 con 0x5
792d00
-10> 2019-05-20 15:34:43.101004 b2efdc30 5 --
192.168.130.132:0/2049423493 >> 192.168.1
30.131:6789/0 conn(0x5792d00 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=45 cs=1
l=1). rx mon.0 seq 6 0x55aaa80 mon_map magic: 0 v1
-9> 2019-05-20 15:34:43.101162 b6714230 1 mgr send_beacon standby
-8> 2019-05-20 15:34:43.101575 b2efdc30 5 --
192.168.130.132:0/2049423493 >> 192.168.1
30.131:6789/0 conn(0x5792d00 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=45 cs=1
l=1). rx mon.0 seq 7 0x55a0540 auth_reply(proto 2 0 (0) Success) v1
-7> 2019-05-20 15:34:43.101889 b2efdc30 5 --
192.168.130.132:0/2049423493 >> 192.168.1
30.131:6789/0 conn(0x5792d00 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=45 cs=1
l=1). rx mon.0 seq 8 0x5590d00 osd_map(42..42 src has 1..42) v3
-6> 2019-05-20 15:34:43.102775 b6714230 10 monclient: _send_mon_message
to mon.odroid-b
at 192.168.130.131:6789/0
-5> 2019-05-20 15:34:43.102838 b6714230 1 --
192.168.130.132:0/2049423493 --> 192.168.
130.131:6789/0 -- mgrbeacon
mgr.odroid-c(75cb9a2d-673b-4a32-897a-05470a08ed58,24139, -, 0)
v6 -- 0x5562400 con 0
-4> 2019-05-20 15:34:43.102991 b6714230 4 mgr init Complete.
-3> 2019-05-20 15:34:43.103065 b06f8c30 4 mgr ms_dispatch standby
mgrmap(e 99) v1
-2> 2019-05-20 15:34:43.103110 b06f8c30 4 mgr handle_mgr_map received
map epoch 99
-1> 2019-05-20 15:34:43.103128 b06f8c30 4 mgr handle_mgr_map active in
map: 0 active i
s 24134
0> 2019-05-20 15:34:43.124462 b06f8c30 -1 *** Caught signal
(Segmentation fault) **
in thread b06f8c30 thread_name:ms_dispatch
ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous
(stable)
1: (()+0x30133c) [0x77033c]
2: (()+0x25750) [0xb688a750]
3: (_ULarm_step()+0x55) [0xb6816ce6]
4: (()+0x255e8) [0xb6cd85e8]
5: (GetStackTrace(void**, int, int)+0x25) [0xb6cd8a3e]
6: (tcmalloc::PageHeap::GrowHeap(unsigned int)+0xb9) [0xb6ccd36a]
7: (tcmalloc::PageHeap::New(unsigned int)+0x79) [0xb6ccd5e6]
8: (tcmalloc::CentralFreeList::Populate()+0x71) [0xb6ccc5ce]
9: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**,
void**)+0x1b) [0xb6ccc76
0]
10: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x6d)
[0xb6ccc7de]
11: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, unsigned
int)+0x51) [0xb6c
cea56]
12: (malloc()+0x22d) [0xb6cd9a8e]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this
.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mgr.odroid-c.log
--- end dump of recent events ---
Kind regards
Jesper
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
