Re: [ceph-users] Noob question - ceph-mgr crash on arm

2019-05-20 Thread Torben Hørup 
Hi

Tcmalloc on arm7 is problematic. You need to compile your own with either 
jemalloc or just libc malloc

/Torben

Den 20. maj 2019 17.48.40 CEST, "Jesper Taxbøl"  skrev:
>I am trying to setup a Ceph cluster on 4 odroid-hc2 instances on top of
>Ubuntu 18.04.
>
>My ceph-mgr deamon keeps crashing on me.
>
>Any advise on how to proceed?
>
>Log on mgr node says something about ms_dispatch:
>
>2019-05-20 15:34:43.070424 b6714230  0 set uid:gid to 64045:64045
>(ceph:ceph)
>2019-05-20 15:34:43.070455 b6714230  0 ceph version 12.2.11
>(26dc3775efc7bb286a1d6d66faee0b
>a30ea23eee) luminous (stable), process ceph-mgr, pid 1169
>2019-05-20 15:34:43.070799 b6714230  0 pidfile_write: ignore empty
>--pid-file
>2019-05-20 15:34:43.101162 b6714230  1 mgr send_beacon standby
>2019-05-20 15:34:43.124462 b06f8c30 -1 *** Caught signal (Segmentation
>fault) **
>in thread b06f8c30 thread_name:ms_dispatch
>
>ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee)
>luminous
>(stable)
>1: (()+0x30133c) [0x77033c]
>2: (()+0x25750) [0xb688a750]
>3: (_ULarm_step()+0x55) [0xb6816ce6]
>4: (()+0x255e8) [0xb6cd85e8]
>5: (GetStackTrace(void**, int, int)+0x25) [0xb6cd8a3e]
>6: (tcmalloc::PageHeap::GrowHeap(unsigned int)+0xb9) [0xb6ccd36a]
>7: (tcmalloc::PageHeap::New(unsigned int)+0x79) [0xb6ccd5e6]
>8: (tcmalloc::CentralFreeList::Populate()+0x71) [0xb6ccc5ce]
>9: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**,
>void**)+0x1b) [0xb6ccc76
>0]
>10: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x6d)
>[0xb6ccc7de]
>11: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int,
>unsigned
>int)+0x51) [0xb6c
>cea56]
>12: (malloc()+0x22d) [0xb6cd9a8e]
>NOTE: a copy of the executable, or `objdump -rdS ` is
>needed to
>interpret this
>.
>
>--- begin dump of recent events ---
>  -90> 2019-05-20 15:34:43.053293 b6714230  5 asok(0x55b5320)
>register_command perfcounter
>s_dump hook 0x554c088
>  -89> 2019-05-20 15:34:43.053322 b6714230  5 asok(0x55b5320)
>register_command 1 hook 0x55
>4c088
>  -88> 2019-05-20 15:34:43.053330 b6714230  5 asok(0x55b5320)
>register_command perf dump h
>ook 0x554c088
>  -87> 2019-05-20 15:34:43.053341 b6714230  5 asok(0x55b5320)
>register_command perfcounter
>s_schema hook 0x554c088
>  -86> 2019-05-20 15:34:43.053360 b6714230  5 asok(0x55b5320)
>register_command perf histog
>ram dump hook 0x554c088
>  -85> 2019-05-20 15:34:43.053374 b6714230  5 asok(0x55b5320)
>register_command 2 hook 0x55
>4c088
>  -84> 2019-05-20 15:34:43.053381 b6714230  5 asok(0x55b5320)
>register_command perf schema
>hook 0x554c088
>  -83> 2019-05-20 15:34:43.053389 b6714230  5 asok(0x55b5320)
>register_command perf histog
>ram schema hook 0x554c088
>  -82> 2019-05-20 15:34:43.053410 b6714230  5 asok(0x55b5320)
>register_command perf reset
>hook 0x554c088
>  -81> 2019-05-20 15:34:43.053418 b6714230  5 asok(0x55b5320)
>register_command config show
>hook 0x554c088
>  -80> 2019-05-20 15:34:43.053425 b6714230  5 asok(0x55b5320)
>register_command config help
>hook 0x554c088
>  -79> 2019-05-20 15:34:43.053436 b6714230  5 asok(0x55b5320)
>register_command config set
>hook 0x554c088
>  -78> 2019-05-20 15:34:43.053444 b6714230  5 asok(0x55b5320)
>register_command config get
>hook 0x554c088
>  -77> 2019-05-20 15:34:43.053459 b6714230  5 asok(0x55b5320)
>register_command config diff
>hook 0x554c088
>  -76> 2019-05-20 15:34:43.053467 b6714230  5 asok(0x55b5320)
>register_command config diff
>get hook 0x554c088
>  -75> 2019-05-20 15:34:43.053475 b6714230  5 asok(0x55b5320)
>register_command log flush h
>ook 0x554c088
>  -74> 2019-05-20 15:34:43.053482 b6714230  5 asok(0x55b5320)
>register_command log dump ho
>ok 0x554c088
>  -73> 2019-05-20 15:34:43.053490 b6714230  5 asok(0x55b5320)
>register_command log reopen
>hook 0x554c088
>  -72> 2019-05-20 15:34:43.053513 b6714230  5 asok(0x55b5320)
>register_command dump_mempoo
>ls hook 0x56e3504
> -71> 2019-05-20 15:34:43.070424 b6714230  0 set uid:gid to 64045:64045
>(ceph:ceph)
>  -70> 2019-05-20 15:34:43.070455 b6714230  0 ceph version 12.2.11
>(26dc3775efc7bb286a1d6d
>66faee0ba30ea23eee) luminous (stable), process ceph-mgr, pid 1169
>-69> 2019-05-20 15:34:43.070799 b6714230  0 pidfile_write: ignore empty
>--pid-file
>  -68> 2019-05-20 15:34:43.074441 b6714230  5 asok(0x55b5320) init
>/var/run/ceph/ceph-mgr.
>odroid-c.asok
>  -67> 2019-05-20 15:34:43.074473 b6714230  5 asok(0x55b5320)
>bind_and_listen /var/run/cep
>h/ceph-mgr.odroid-c.asok
>  -66> 2019-05-20 15:34:43.074615 b6714230  5 asok(0x55b5320)
>register_command 0 hook 0x55
>4c1d0
>  -65> 2019-05-20 15:34:43.074633 b6714230  5 asok(0x55b5320)
>register_command version hoo
>k 0x554c1d0
>  -64> 2019-05-20 15:34:43.074654 b6714230  5 asok(0x55b5320)
>register_command git_version
>hook 0x554c1d0
>  -63> 2019-05-20 15:34:43.074674 b6714230  5 asok(0x55b5320)
>register_command help hook 0
>x554c1d8
>  -62> 2019-05-20 15:34:43.074694 b6714230  5 asok(0x55b5320)
>register_command get_command
>_descriptions hook 0x554c1e0
>-61>

[ceph-users] Noob question - ceph-mgr crash on arm

2019-05-20 Thread Jesper Taxbøl 
I am trying to setup a Ceph cluster on 4 odroid-hc2 instances on top of
Ubuntu 18.04.

My ceph-mgr deamon keeps crashing on me.

Any advise on how to proceed?

Log on mgr node says something about ms_dispatch:

2019-05-20 15:34:43.070424 b6714230  0 set uid:gid to 64045:64045
(ceph:ceph)
2019-05-20 15:34:43.070455 b6714230  0 ceph version 12.2.11
(26dc3775efc7bb286a1d6d66faee0b
a30ea23eee) luminous (stable), process ceph-mgr, pid 1169
2019-05-20 15:34:43.070799 b6714230  0 pidfile_write: ignore empty
--pid-file
2019-05-20 15:34:43.101162 b6714230  1 mgr send_beacon standby
2019-05-20 15:34:43.124462 b06f8c30 -1 *** Caught signal (Segmentation
fault) **
in thread b06f8c30 thread_name:ms_dispatch

ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous
(stable)
1: (()+0x30133c) [0x77033c]
2: (()+0x25750) [0xb688a750]
3: (_ULarm_step()+0x55) [0xb6816ce6]
4: (()+0x255e8) [0xb6cd85e8]
5: (GetStackTrace(void**, int, int)+0x25) [0xb6cd8a3e]
6: (tcmalloc::PageHeap::GrowHeap(unsigned int)+0xb9) [0xb6ccd36a]
7: (tcmalloc::PageHeap::New(unsigned int)+0x79) [0xb6ccd5e6]
8: (tcmalloc::CentralFreeList::Populate()+0x71) [0xb6ccc5ce]
9: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**,
void**)+0x1b) [0xb6ccc76
0]
10: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x6d)
[0xb6ccc7de]
11: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, unsigned
int)+0x51) [0xb6c
cea56]
12: (malloc()+0x22d) [0xb6cd9a8e]
NOTE: a copy of the executable, or `objdump -rdS ` is needed to
interpret this
.

--- begin dump of recent events ---
  -90> 2019-05-20 15:34:43.053293 b6714230  5 asok(0x55b5320)
register_command perfcounter
s_dump hook 0x554c088
  -89> 2019-05-20 15:34:43.053322 b6714230  5 asok(0x55b5320)
register_command 1 hook 0x55
4c088
  -88> 2019-05-20 15:34:43.053330 b6714230  5 asok(0x55b5320)
register_command perf dump h
ook 0x554c088
  -87> 2019-05-20 15:34:43.053341 b6714230  5 asok(0x55b5320)
register_command perfcounter
s_schema hook 0x554c088
  -86> 2019-05-20 15:34:43.053360 b6714230  5 asok(0x55b5320)
register_command perf histog
ram dump hook 0x554c088
  -85> 2019-05-20 15:34:43.053374 b6714230  5 asok(0x55b5320)
register_command 2 hook 0x55
4c088
  -84> 2019-05-20 15:34:43.053381 b6714230  5 asok(0x55b5320)
register_command perf schema
hook 0x554c088
  -83> 2019-05-20 15:34:43.053389 b6714230  5 asok(0x55b5320)
register_command perf histog
ram schema hook 0x554c088
  -82> 2019-05-20 15:34:43.053410 b6714230  5 asok(0x55b5320)
register_command perf reset
hook 0x554c088
  -81> 2019-05-20 15:34:43.053418 b6714230  5 asok(0x55b5320)
register_command config show
hook 0x554c088
  -80> 2019-05-20 15:34:43.053425 b6714230  5 asok(0x55b5320)
register_command config help
hook 0x554c088
  -79> 2019-05-20 15:34:43.053436 b6714230  5 asok(0x55b5320)
register_command config set
hook 0x554c088
  -78> 2019-05-20 15:34:43.053444 b6714230  5 asok(0x55b5320)
register_command config get
hook 0x554c088
  -77> 2019-05-20 15:34:43.053459 b6714230  5 asok(0x55b5320)
register_command config diff
hook 0x554c088
  -76> 2019-05-20 15:34:43.053467 b6714230  5 asok(0x55b5320)
register_command config diff
get hook 0x554c088
  -75> 2019-05-20 15:34:43.053475 b6714230  5 asok(0x55b5320)
register_command log flush h
ook 0x554c088
  -74> 2019-05-20 15:34:43.053482 b6714230  5 asok(0x55b5320)
register_command log dump ho
ok 0x554c088
  -73> 2019-05-20 15:34:43.053490 b6714230  5 asok(0x55b5320)
register_command log reopen
hook 0x554c088
  -72> 2019-05-20 15:34:43.053513 b6714230  5 asok(0x55b5320)
register_command dump_mempoo
ls hook 0x56e3504
  -71> 2019-05-20 15:34:43.070424 b6714230  0 set uid:gid to 64045:64045
(ceph:ceph)
  -70> 2019-05-20 15:34:43.070455 b6714230  0 ceph version 12.2.11
(26dc3775efc7bb286a1d6d
66faee0ba30ea23eee) luminous (stable), process ceph-mgr, pid 1169
  -69> 2019-05-20 15:34:43.070799 b6714230  0 pidfile_write: ignore empty
--pid-file
  -68> 2019-05-20 15:34:43.074441 b6714230  5 asok(0x55b5320) init
/var/run/ceph/ceph-mgr.
odroid-c.asok
  -67> 2019-05-20 15:34:43.074473 b6714230  5 asok(0x55b5320)
bind_and_listen /var/run/cep
h/ceph-mgr.odroid-c.asok
  -66> 2019-05-20 15:34:43.074615 b6714230  5 asok(0x55b5320)
register_command 0 hook 0x55
4c1d0
  -65> 2019-05-20 15:34:43.074633 b6714230  5 asok(0x55b5320)
register_command version hoo
k 0x554c1d0
  -64> 2019-05-20 15:34:43.074654 b6714230  5 asok(0x55b5320)
register_command git_version
hook 0x554c1d0
  -63> 2019-05-20 15:34:43.074674 b6714230  5 asok(0x55b5320)
register_command help hook 0
x554c1d8
  -62> 2019-05-20 15:34:43.074694 b6714230  5 asok(0x55b5320)
register_command get_command
_descriptions hook 0x554c1e0
  -61> 2019-05-20 15:34:43.074785 b3effc30  5 asok(0x55b5320) entry start
  -60> 2019-05-20 15:34:43.076464 b36fec30  2 Event(0x554e068 nevent=5000
time_id=1).set_o
wner idx=0 owner=3010456624
  -59> 2019-05-20 15:34:43.076559 b2efdc30  2 Event(0x554e488 nevent=5000
time_id=1).set_o
wner idx=1 owner=3002063920