I'm on Luminous 12.1.1 and noticed I have flapping OSDs. Even with `ceph osd set nodown`, the OSDs will catch signal Aborted and sometimes Segmentation fault 2-5 minutes after starting. I verified hosts can talk to eachother on the cluster network. I've rebooted the hosts. I'm running out of ideas. Please advise.
Tally of crashes: roger@osd1:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog{,.1} | awk '{print $9}' | sort | uniq -c | sort -nr 100 (Segmentation 77 (Aborted) roger@osd2:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog{,.1} | awk '{print $9}' | sort | uniq -c | sort -nr 77 (Aborted) 13 (Segmentation roger@osd3:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog{,.1} | awk '{print $9}' | sort | uniq -c | sort -nr 86 (Aborted) 3 (Segmentation First crash observed Jul 19: roger@osd1:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog.1 | head -1 Jul 19 10:07:12 osd1 ceph-osd[13491]: *** Caught signal (Aborted) ** roger@osd2:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog.1 | head -1 Jul 19 10:07:36 osd2 ceph-osd[13937]: *** Caught signal (Aborted) ** roger@osd3:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog.1 | head -1 Jul 19 16:07:12 osd3 ceph-osd[8807]: *** Caught signal (Aborted) ** Crashes started with Luminous 12.1.0: roger@osd1:~$ sudo grep 'Jul 19 10:07:12.*ceph version' /var/log/syslog.1 | head -1 Jul 19 10:07:12 osd1 ceph-osd[13491]: ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev) roger@osd2:~$ sudo grep 'Jul 19 10:07:36.*ceph version' /var/log/syslog.1 | head -1 Jul 19 10:07:36 osd2 ceph-osd[13937]: ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev) roger@osd3:~$ sudo grep 'Jul 19 16:07:12.*ceph version' /var/log/syslog.1 | head -1 Jul 19 16:07:12 osd3 ceph-osd[8807]: ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev) Representative example from osd1 logs: Jul 20 13:42:18 osd1 ceph-osd[4035]: *** Caught signal (Segmentation fault) ** Jul 20 13:42:18 osd1 ceph-osd[4035]: in thread 7f52960e7700 thread_name:msgr-worker-2 Jul 20 13:42:18 osd1 ceph-osd[4035]: 2017-07-20 13:42:18.658076 7f529bf85c80 -1 osd.3 3444 log_to_monitors {default=true} Jul 20 13:42:18 osd1 ceph-osd[4035]: 2017-07-20 13:42:18.662695 7f52968e8700 -1 failed to decode message of type 70 v3: buffer::malformed_input: void osd_peer_stat_t::decode(ceph::buffer::list::iterator&) no longer understand old encoding version 1 < struct_compat Jul 20 13:42:18 osd1 ceph-osd[4035]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:42:18 osd1 ceph-osd[4035]: 1: (()+0xa257a4) [0x55bc98fe27a4] Jul 20 13:42:18 osd1 ceph-osd[4035]: 2: (()+0x11390) [0x7f529a468390] Jul 20 13:42:18 osd1 ceph-osd[4035]: 3: (cephx_verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list::iterator&, CephXServiceTicketInfo&, ceph::buffer::list&)+0x496) [0x55bc991b0ca6] Jul 20 13:42:18 osd1 ceph-osd[4035]: 4: (CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list&, ceph::buffer::list&, EntityName&, unsigned long&, AuthCapsInfo&, CryptoKey&, unsigned long*)+0x31a) [0x55bc991a2cda] Jul 20 13:42:18 osd1 ceph-osd[4035]: 5: (OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&, ceph::buffer::list&, bool&, CryptoKey&)+0xf9) [0x55bc98a2c759] Jul 20 13:42:18 osd1 ceph-osd[4035]: 6: (AsyncConnection::handle_connect_msg(ceph_msg_connect&, ceph::buffer::list&, ceph::buffer::list&)+0x228) [0x55bc99271108] Jul 20 13:42:18 osd1 ceph-osd[4035]: 7: (AsyncConnection::_process_connection()+0x1e07) [0x55bc99276a57] Jul 20 13:42:18 osd1 ceph-osd[4035]: 8: (AsyncConnection::process()+0x1ae8) [0x55bc9927b978] Jul 20 13:42:18 osd1 ceph-osd[4035]: 9: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55bc990c6148] Jul 20 13:42:18 osd1 ceph-osd[4035]: 10: (()+0xb0d0d8) [0x55bc990ca0d8] Jul 20 13:42:18 osd1 ceph-osd[4035]: 11: (()+0xb8c80) [0x7f5299d6fc80] Jul 20 13:42:18 osd1 ceph-osd[4035]: 12: (()+0x76ba) [0x7f529a45e6ba] Jul 20 13:42:18 osd1 ceph-osd[4035]: 13: (clone()+0x6d) [0x7f52994d53dd] Jul 20 13:42:18 osd1 ceph-osd[4035]: 2017-07-20 13:42:18.662763 7f52960e7700 -1 *** Caught signal (Segmentation fault) ** Jul 20 13:42:18 osd1 ceph-osd[4035]: in thread 7f52960e7700 thread_name:msgr-worker-2 Jul 20 13:42:18 osd1 ceph-osd[4035]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:42:18 osd1 ceph-osd[4035]: 1: (()+0xa257a4) [0x55bc98fe27a4] Jul 20 13:42:18 osd1 ceph-osd[4035]: 2: (()+0x11390) [0x7f529a468390] Jul 20 13:42:18 osd1 ceph-osd[4035]: 3: (cephx_verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list::iterator&, CephXServiceTicketInfo&, ceph::buffer::list&)+0x496) [0x55bc991b0ca6] Jul 20 13:42:18 osd1 ceph-osd[4035]: 4: (CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list&, ceph::buffer::list&, EntityName&, unsigned long&, AuthCapsInfo&, CryptoKey&, unsigned long*)+0x31a) [0x55bc991a2cda] Jul 20 13:42:18 osd1 ceph-osd[4035]: 5: (OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&, ceph::buffer::list&, bool&, CryptoKey&)+0xf9) [0x55bc98a2c759] Jul 20 13:42:18 osd1 ceph-osd[4035]: 6: (AsyncConnection::handle_connect_msg(ceph_msg_connect&, ceph::buffer::list&, ceph::buffer::list&)+0x228) [0x55bc99271108] Jul 20 13:42:18 osd1 ceph-osd[4035]: 7: (AsyncConnection::_process_connection()+0x1e07) [0x55bc99276a57] Jul 20 13:42:18 osd1 ceph-osd[4035]: 8: (AsyncConnection::process()+0x1ae8) [0x55bc9927b978] Jul 20 13:42:18 osd1 ceph-osd[4035]: 9: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55bc990c6148] Jul 20 13:42:18 osd1 ceph-osd[4035]: 10: (()+0xb0d0d8) [0x55bc990ca0d8] Jul 20 13:42:18 osd1 ceph-osd[4035]: 11: (()+0xb8c80) [0x7f5299d6fc80] Jul 20 13:42:18 osd1 ceph-osd[4035]: 12: (()+0x76ba) [0x7f529a45e6ba] Jul 20 13:42:18 osd1 ceph-osd[4035]: 13: (clone()+0x6d) [0x7f52994d53dd] Jul 20 13:42:18 osd1 ceph-osd[4035]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:42:18 osd1 ceph-osd[4035]: -18> 2017-07-20 13:42:18.658076 7f529bf85c80 -1 osd.3 3444 log_to_monitors {default=true} Jul 20 13:42:18 osd1 ceph-osd[4035]: -5> 2017-07-20 13:42:18.662695 7f52968e8700 -1 failed to decode message of type 70 v3: buffer::malformed_input: void osd_peer_stat_t::decode(ceph::buffer::list::iterator&) no longer understand old encoding version 1 < struct_compat Jul 20 13:42:18 osd1 ceph-osd[4035]: 0> 2017-07-20 13:42:18.662763 7f52960e7700 -1 *** Caught signal (Segmentation fault) ** Jul 20 13:42:18 osd1 ceph-osd[4035]: in thread 7f52960e7700 thread_name:msgr-worker-2 Jul 20 13:42:18 osd1 ceph-osd[4035]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:42:18 osd1 ceph-osd[4035]: 1: (()+0xa257a4) [0x55bc98fe27a4] Jul 20 13:42:18 osd1 ceph-osd[4035]: 2: (()+0x11390) [0x7f529a468390] Jul 20 13:42:18 osd1 ceph-osd[4035]: 3: (cephx_verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list::iterator&, CephXServiceTicketInfo&, ceph::buffer::list&)+0x496) [0x55bc991b0ca6] Jul 20 13:42:18 osd1 ceph-osd[4035]: 4: (CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list&, ceph::buffer::list&, EntityName&, unsigned long&, AuthCapsInfo&, CryptoKey&, unsigned long*)+0x31a) [0x55bc991a2cda] Jul 20 13:42:18 osd1 ceph-osd[4035]: 5: (OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&, ceph::buffer::list&, bool&, CryptoKey&)+0xf9) [0x55bc98a2c759] Jul 20 13:42:18 osd1 ceph-osd[4035]: 6: (AsyncConnection::handle_connect_msg(ceph_msg_connect&, ceph::buffer::list&, ceph::buffer::list&)+0x228) [0x55bc99271108] Jul 20 13:42:18 osd1 ceph-osd[4035]: 7: (AsyncConnection::_process_connection()+0x1e07) [0x55bc99276a57] Jul 20 13:42:18 osd1 ceph-osd[4035]: 8: (AsyncConnection::process()+0x1ae8) [0x55bc9927b978] Jul 20 13:42:18 osd1 ceph-osd[4035]: 9: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55bc990c6148] Jul 20 13:42:18 osd1 ceph-osd[4035]: 10: (()+0xb0d0d8) [0x55bc990ca0d8] Jul 20 13:42:18 osd1 ceph-osd[4035]: 11: (()+0xb8c80) [0x7f5299d6fc80] Jul 20 13:42:18 osd1 ceph-osd[4035]: 12: (()+0x76ba) [0x7f529a45e6ba] Jul 20 13:42:18 osd1 ceph-osd[4035]: 13: (clone()+0x6d) [0x7f52994d53dd] Jul 20 13:42:18 osd1 ceph-osd[4035]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:42:18 osd1 systemd[1]: ceph-osd@3.service: Main process exited, code=killed, status=11/SEGV Jul 20 13:42:18 osd1 systemd[1]: ceph-osd@3.service: Unit entered failed state. Jul 20 13:42:18 osd1 systemd[1]: ceph-osd@3.service: Failed with result 'signal'. Jul 20 13:42:38 osd1 systemd[1]: ceph-osd@3.service: Service hold-off time over, scheduling restart. Jul 20 13:42:38 osd1 systemd[1]: Stopped Ceph object storage daemon osd.3. Jul 20 13:42:38 osd1 systemd[1]: Starting Ceph object storage daemon osd.3... Jul 20 13:42:39 osd1 systemd[1]: Started Ceph object storage daemon osd.3. Jul 20 13:42:39 osd1 ceph-osd[4130]: starting osd.3 at - osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal Jul 20 13:43:02 osd1 sshd[3497]: Received disconnect from 192.168.0.7 port 55258:11: disconnected by user Jul 20 13:43:02 osd1 sshd[3497]: Disconnected from 192.168.0.7 port 55258 Jul 20 13:43:02 osd1 sshd[3466]: pam_unix(sshd:session): session closed for user roger Jul 20 13:43:02 osd1 systemd-logind[1393]: Removed session 10. Jul 20 13:44:53 osd1 ceph-osd[4130]: 2017-07-20 13:44:53.540934 7f303995dc80 -1 osd.3 3444 log_to_monitors {default=true} Jul 20 13:45:33 osd1 ceph-osd[4130]: 2017-07-20 13:45:33.544688 7f30302de700 -1 osd.3 3458 heartbeat_check: no reply from 192.168.0.26:6801 osd.0 since back 2017-07-20 13:45:12.643355 front 2017-07-20 13:45:12.643355 (cutoff 2017-07-20 13:45:13.544686) Jul 20 13:46:01 osd1 ceph-osd[4130]: /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*, char**, uint32_t*)' thread 7f3033abf700 time 2017-07-20 13:46:01.429584 Jul 20 13:46:01 osd1 ceph-osd[4130]: /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end) Jul 20 13:46:01 osd1 ceph-osd[4130]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:46:01 osd1 ceph-osd[4130]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55f2073ffb72] Jul 20 13:46:01 osd1 ceph-osd[4130]: 2: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9] Jul 20 13:46:01 osd1 ceph-osd[4130]: 3: (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce] Jul 20 13:46:01 osd1 ceph-osd[4130]: 4: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x55f20743a5d8] Jul 20 13:46:01 osd1 ceph-osd[4130]: 5: (AsyncConnection::process()+0x1d4e) [0x55f207656bde] Jul 20 13:46:01 osd1 ceph-osd[4130]: 6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148] Jul 20 13:46:01 osd1 ceph-osd[4130]: 7: (()+0xb0d0d8) [0x55f2074a50d8] Jul 20 13:46:01 osd1 ceph-osd[4130]: 8: (()+0xb8c80) [0x7f3037747c80] Jul 20 13:46:01 osd1 ceph-osd[4130]: 9: (()+0x76ba) [0x7f3037e366ba] Jul 20 13:46:01 osd1 ceph-osd[4130]: 10: (clone()+0x6d) [0x7f3036ead3dd] Jul 20 13:46:01 osd1 ceph-osd[4130]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:46:01 osd1 ceph-osd[4130]: 2017-07-20 13:46:01.434169 7f3033abf700 -1 /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*, char**, uint32_t*)' thread 7f3033abf700 time 2017-07-20 13:46:01.429584 Jul 20 13:46:01 osd1 ceph-osd[4130]: /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end) Jul 20 13:46:01 osd1 ceph-osd[4130]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:46:01 osd1 ceph-osd[4130]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55f2073ffb72] Jul 20 13:46:01 osd1 ceph-osd[4130]: 2: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9] Jul 20 13:46:01 osd1 ceph-osd[4130]: 3: (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce] Jul 20 13:46:01 osd1 ceph-osd[4130]: 4: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x55f20743a5d8] Jul 20 13:46:01 osd1 ceph-osd[4130]: 5: (AsyncConnection::process()+0x1d4e) [0x55f207656bde] Jul 20 13:46:01 osd1 ceph-osd[4130]: 6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148] Jul 20 13:46:01 osd1 ceph-osd[4130]: 7: (()+0xb0d0d8) [0x55f2074a50d8] Jul 20 13:46:01 osd1 ceph-osd[4130]: 8: (()+0xb8c80) [0x7f3037747c80] Jul 20 13:46:01 osd1 ceph-osd[4130]: 9: (()+0x76ba) [0x7f3037e366ba] Jul 20 13:46:01 osd1 ceph-osd[4130]: 10: (clone()+0x6d) [0x7f3036ead3dd] Jul 20 13:46:01 osd1 ceph-osd[4130]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:46:01 osd1 ceph-osd[4130]: 0> 2017-07-20 13:46:01.434169 7f3033abf700 -1 /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*, char**, uint32_t*)' thread 7f3033abf700 time 2017-07-20 13:46:01.429584 Jul 20 13:46:01 osd1 ceph-osd[4130]: /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end) Jul 20 13:46:01 osd1 ceph-osd[4130]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:46:01 osd1 ceph-osd[4130]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55f2073ffb72] Jul 20 13:46:01 osd1 ceph-osd[4130]: 2: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9] Jul 20 13:46:01 osd1 ceph-osd[4130]: 3: (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce] Jul 20 13:46:01 osd1 ceph-osd[4130]: 4: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x55f20743a5d8] Jul 20 13:46:01 osd1 ceph-osd[4130]: 5: (AsyncConnection::process()+0x1d4e) [0x55f207656bde] Jul 20 13:46:01 osd1 ceph-osd[4130]: 6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148] Jul 20 13:46:01 osd1 ceph-osd[4130]: 7: (()+0xb0d0d8) [0x55f2074a50d8] Jul 20 13:46:01 osd1 ceph-osd[4130]: 8: (()+0xb8c80) [0x7f3037747c80] Jul 20 13:46:01 osd1 ceph-osd[4130]: 9: (()+0x76ba) [0x7f3037e366ba] Jul 20 13:46:01 osd1 ceph-osd[4130]: 10: (clone()+0x6d) [0x7f3036ead3dd] Jul 20 13:46:01 osd1 ceph-osd[4130]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:46:01 osd1 ceph-osd[4130]: *** Caught signal (Aborted) ** Jul 20 13:46:01 osd1 ceph-osd[4130]: in thread 7f3033abf700 thread_name:msgr-worker-2 Jul 20 13:46:01 osd1 ceph-osd[4130]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:46:01 osd1 ceph-osd[4130]: 1: (()+0xa257a4) [0x55f2073bd7a4] Jul 20 13:46:01 osd1 ceph-osd[4130]: 2: (()+0x11390) [0x7f3037e40390] Jul 20 13:46:01 osd1 ceph-osd[4130]: 3: (gsignal()+0x38) [0x7f3036ddb428] Jul 20 13:46:01 osd1 ceph-osd[4130]: 4: (abort()+0x16a) [0x7f3036ddd02a] Jul 20 13:46:01 osd1 ceph-osd[4130]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55f2073ffcfe] Jul 20 13:46:01 osd1 ceph-osd[4130]: 6: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9] Jul 20 13:46:01 osd1 ceph-osd[4130]: 7: (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce] Jul 20 13:46:01 osd1 ceph-osd[4130]: 8: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x55f20743a5d8] Jul 20 13:46:01 osd1 ceph-osd[4130]: 9: (AsyncConnection::process()+0x1d4e) [0x55f207656bde] Jul 20 13:46:01 osd1 ceph-osd[4130]: 10: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148] Jul 20 13:46:01 osd1 ceph-osd[4130]: 11: (()+0xb0d0d8) [0x55f2074a50d8] Jul 20 13:46:01 osd1 ceph-osd[4130]: 12: (()+0xb8c80) [0x7f3037747c80] Jul 20 13:46:01 osd1 ceph-osd[4130]: 13: (()+0x76ba) [0x7f3037e366ba] Jul 20 13:46:01 osd1 ceph-osd[4130]: 14: (clone()+0x6d) [0x7f3036ead3dd] Jul 20 13:46:01 osd1 ceph-osd[4130]: 2017-07-20 13:46:01.481468 7f3033abf700 -1 *** Caught signal (Aborted) ** Jul 20 13:46:01 osd1 ceph-osd[4130]: in thread 7f3033abf700 thread_name:msgr-worker-2 Jul 20 13:46:01 osd1 ceph-osd[4130]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:46:01 osd1 ceph-osd[4130]: 1: (()+0xa257a4) [0x55f2073bd7a4] Jul 20 13:46:01 osd1 ceph-osd[4130]: 2: (()+0x11390) [0x7f3037e40390] Jul 20 13:46:01 osd1 ceph-osd[4130]: 3: (gsignal()+0x38) [0x7f3036ddb428] Jul 20 13:46:01 osd1 ceph-osd[4130]: 4: (abort()+0x16a) [0x7f3036ddd02a] Jul 20 13:46:01 osd1 ceph-osd[4130]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55f2073ffcfe] Jul 20 13:46:01 osd1 ceph-osd[4130]: 6: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9] Jul 20 13:46:01 osd1 ceph-osd[4130]: 7: (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce] Jul 20 13:46:01 osd1 ceph-osd[4130]: 8: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x55f20743a5d8] Jul 20 13:46:01 osd1 ceph-osd[4130]: 9: (AsyncConnection::process()+0x1d4e) [0x55f207656bde] Jul 20 13:46:01 osd1 ceph-osd[4130]: 10: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148] Jul 20 13:46:01 osd1 ceph-osd[4130]: 11: (()+0xb0d0d8) [0x55f2074a50d8] Jul 20 13:46:01 osd1 ceph-osd[4130]: 12: (()+0xb8c80) [0x7f3037747c80] Jul 20 13:46:01 osd1 ceph-osd[4130]: 13: (()+0x76ba) [0x7f3037e366ba] Jul 20 13:46:01 osd1 ceph-osd[4130]: 14: (clone()+0x6d) [0x7f3036ead3dd] Jul 20 13:46:01 osd1 ceph-osd[4130]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:46:01 osd1 ceph-osd[4130]: 0> 2017-07-20 13:46:01.481468 7f3033abf700 -1 *** Caught signal (Aborted) ** Jul 20 13:46:01 osd1 ceph-osd[4130]: in thread 7f3033abf700 thread_name:msgr-worker-2 Jul 20 13:46:01 osd1 ceph-osd[4130]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:46:01 osd1 ceph-osd[4130]: 1: (()+0xa257a4) [0x55f2073bd7a4] Jul 20 13:46:01 osd1 ceph-osd[4130]: 2: (()+0x11390) [0x7f3037e40390] Jul 20 13:46:01 osd1 ceph-osd[4130]: 3: (gsignal()+0x38) [0x7f3036ddb428] Jul 20 13:46:01 osd1 ceph-osd[4130]: 4: (abort()+0x16a) [0x7f3036ddd02a] Jul 20 13:46:01 osd1 ceph-osd[4130]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55f2073ffcfe] Jul 20 13:46:01 osd1 ceph-osd[4130]: 6: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9] Jul 20 13:46:01 osd1 ceph-osd[4130]: 7: (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce] Jul 20 13:46:01 osd1 ceph-osd[4130]: 8: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x55f20743a5d8] Jul 20 13:46:01 osd1 ceph-osd[4130]: 9: (AsyncConnection::process()+0x1d4e) [0x55f207656bde] Jul 20 13:46:01 osd1 ceph-osd[4130]: 10: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148] Jul 20 13:46:01 osd1 ceph-osd[4130]: 11: (()+0xb0d0d8) [0x55f2074a50d8] Jul 20 13:46:01 osd1 ceph-osd[4130]: 12: (()+0xb8c80) [0x7f3037747c80] Jul 20 13:46:01 osd1 ceph-osd[4130]: 13: (()+0x76ba) [0x7f3037e366ba] Jul 20 13:46:01 osd1 ceph-osd[4130]: 14: (clone()+0x6d) [0x7f3036ead3dd] Jul 20 13:46:01 osd1 ceph-osd[4130]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:46:01 osd1 systemd[1]: ceph-osd@3.service: Main process exited, code=killed, status=6/ABRT Jul 20 13:46:01 osd1 systemd[1]: ceph-osd@3.service: Unit entered failed state. Jul 20 13:46:01 osd1 systemd[1]: ceph-osd@3.service: Failed with result 'signal'. Jul 20 13:46:21 osd1 systemd[1]: ceph-osd@3.service: Service hold-off time over, scheduling restart. Jul 20 13:46:21 osd1 systemd[1]: Stopped Ceph object storage daemon osd.3. Jul 20 13:46:21 osd1 systemd[1]: Starting Ceph object storage daemon osd.3... Jul 20 13:46:22 osd1 systemd[1]: Started Ceph object storage daemon osd.3. Jul 20 13:46:22 osd1 ceph-osd[4223]: starting osd.3 at - osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal Jul 20 13:48:39 osd1 ceph-osd[4223]: *** Caught signal (Segmentation fault) ** Jul 20 13:48:39 osd1 ceph-osd[4223]: in thread 7f10b1baa700 thread_name:msgr-worker-2 Jul 20 13:48:39 osd1 ceph-osd[4223]: 2017-07-20 13:48:39.470084 7f10b7a48c80 -1 osd.3 3460 log_to_monitors {default=true} Jul 20 13:48:39 osd1 ceph-osd[4223]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:48:39 osd1 ceph-osd[4223]: 1: (()+0xa257a4) [0x55cead5be7a4] Jul 20 13:48:39 osd1 ceph-osd[4223]: 2: (()+0x11390) [0x7f10b5f2b390] Jul 20 13:48:39 osd1 ceph-osd[4223]: 3: (cephx_verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list::iterator&, CephXServiceTicketInfo&, ceph::buffer::list&)+0x496) [0x55cead78cca6] Jul 20 13:48:39 osd1 ceph-osd[4223]: 4: (CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list&, ceph::buffer::list&, EntityName&, unsigned long&, AuthCapsInfo&, CryptoKey&, unsigned long*)+0x31a) [0x55cead77ecda] Jul 20 13:48:39 osd1 ceph-osd[4223]: 5: (OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&, ceph::buffer::list&, bool&, CryptoKey&)+0xf9) [0x55cead008759] Jul 20 13:48:39 osd1 ceph-osd[4223]: 6: (AsyncConnection::handle_connect_msg(ceph_msg_connect&, ceph::buffer::list&, ceph::buffer::list&)+0x228) [0x55cead84d108] Jul 20 13:48:39 osd1 ceph-osd[4223]: 7: (AsyncConnection::_process_connection()+0x1e07) [0x55cead852a57] Jul 20 13:48:39 osd1 ceph-osd[4223]: 7: (AsyncConnection::_process_connection()+0x1e07) [0x55cead852a57] Jul 20 13:48:39 osd1 ceph-osd[4223]: 8: (AsyncConnection::process()+0x1ae8) [0x55cead857978] Jul 20 13:48:39 osd1 ceph-osd[4223]: 9: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55cead6a2148] Jul 20 13:48:39 osd1 ceph-osd[4223]: 10: (()+0xb0d0d8) [0x55cead6a60d8] Jul 20 13:48:39 osd1 ceph-osd[4223]: 11: (()+0xb8c80) [0x7f10b5832c80] Jul 20 13:48:39 osd1 ceph-osd[4223]: 12: (()+0x76ba) [0x7f10b5f216ba] Jul 20 13:48:39 osd1 ceph-osd[4223]: 13: (clone()+0x6d) [0x7f10b4f983dd] Jul 20 13:48:39 osd1 ceph-osd[4223]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:48:39 osd1 systemd[1]: ceph-osd@3.service: Main process exited, code=killed, status=11/SEGV Jul 20 13:48:39 osd1 systemd[1]: ceph-osd@3.service: Unit entered failed state. Jul 20 13:48:39 osd1 systemd[1]: ceph-osd@3.service: Failed with result 'signal'. Jul 20 13:48:59 osd1 systemd[1]: ceph-osd@3.service: Service hold-off time over, scheduling restart. Jul 20 13:48:59 osd1 systemd[1]: Stopped Ceph object storage daemon osd.3. Jul 20 13:48:59 osd1 systemd[1]: Starting Ceph object storage daemon osd.3... Jul 20 13:49:00 osd1 systemd[1]: Started Ceph object storage daemon osd.3. Jul 20 13:49:00 osd1 ceph-osd[4314]: starting osd.3 at - osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal Jul 20 13:51:15 osd1 ceph-osd[4314]: 2017-07-20 13:51:15.595553 7feabcf96c80 -1 osd.3 3460 log_to_monitors {default=true} Jul 20 13:51:54 osd1 ceph-osd[4314]: 2017-07-20 13:51:54.599050 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:34.599047) Jul 20 13:51:55 osd1 ceph-osd[4314]: 2017-07-20 13:51:55.599219 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:35.599217) Jul 20 13:51:56 osd1 ceph-osd[4314]: 2017-07-20 13:51:56.599336 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:36.599335) Jul 20 13:51:57 osd1 ceph-osd[4314]: 2017-07-20 13:51:57.599445 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:37.599443) Jul 20 13:51:58 osd1 ceph-osd[4314]: 2017-07-20 13:51:58.599563 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:38.599562) Jul 20 13:52:26 osd1 ceph-osd[4314]: /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*, char**, uint32_t*)' thread 7feab78f9700 time 2017-07-20 13:52:26.501284 Jul 20 13:52:26 osd1 ceph-osd[4314]: /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end) Jul 20 13:52:26 osd1 ceph-osd[4314]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:52:26 osd1 ceph-osd[4314]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5565a2421b72] Jul 20 13:52:26 osd1 ceph-osd[4314]: 2: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9] Jul 20 13:52:26 osd1 ceph-osd[4314]: 3: (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce] Jul 20 13:52:26 osd1 ceph-osd[4314]: 4: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x5565a245c5d8] Jul 20 13:52:26 osd1 ceph-osd[4314]: 5: (AsyncConnection::process()+0x1d4e) [0x5565a2678bde] Jul 20 13:52:26 osd1 ceph-osd[4314]: 6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148] Jul 20 13:52:26 osd1 ceph-osd[4314]: 7: (()+0xb0d0d8) [0x5565a24c70d8] Jul 20 13:52:26 osd1 ceph-osd[4314]: 8: (()+0xb8c80) [0x7feabad80c80] Jul 20 13:52:26 osd1 ceph-osd[4314]: 9: (()+0x76ba) [0x7feabb46f6ba] Jul 20 13:52:26 osd1 ceph-osd[4314]: 10: (clone()+0x6d) [0x7feaba4e63dd] Jul 20 13:52:26 osd1 ceph-osd[4314]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:52:26 osd1 ceph-osd[4314]: 2017-07-20 13:52:26.505919 7feab78f9700 -1 /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*, char**, uint32_t*)' thread 7feab78f9700 time 2017-07-20 13:52:26.501284 Jul 20 13:52:26 osd1 ceph-osd[4314]: /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end) Jul 20 13:52:26 osd1 ceph-osd[4314]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:52:26 osd1 ceph-osd[4314]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5565a2421b72] Jul 20 13:52:26 osd1 ceph-osd[4314]: 2: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9] Jul 20 13:52:26 osd1 ceph-osd[4314]: 3: (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce] Jul 20 13:52:26 osd1 ceph-osd[4314]: 4: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x5565a245c5d8] Jul 20 13:52:26 osd1 ceph-osd[4314]: 5: (AsyncConnection::process()+0x1d4e) [0x5565a2678bde] Jul 20 13:52:26 osd1 ceph-osd[4314]: 6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148] Jul 20 13:52:26 osd1 ceph-osd[4314]: 7: (()+0xb0d0d8) [0x5565a24c70d8] Jul 20 13:52:26 osd1 ceph-osd[4314]: 8: (()+0xb8c80) [0x7feabad80c80] Jul 20 13:52:26 osd1 ceph-osd[4314]: 9: (()+0x76ba) [0x7feabb46f6ba] Jul 20 13:52:26 osd1 ceph-osd[4314]: 10: (clone()+0x6d) [0x7feaba4e63dd] Jul 20 13:52:26 osd1 ceph-osd[4314]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:52:26 osd1 ceph-osd[4314]: 0> 2017-07-20 13:52:26.505919 7feab78f9700 -1 /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*, char**, uint32_t*)' thread 7feab78f9700 time 2017-07-20 13:52:26.501284 Jul 20 13:52:26 osd1 ceph-osd[4314]: /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end) Jul 20 13:52:26 osd1 ceph-osd[4314]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:52:26 osd1 ceph-osd[4314]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5565a2421b72] Jul 20 13:52:26 osd1 ceph-osd[4314]: 2: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9] Jul 20 13:52:26 osd1 ceph-osd[4314]: 3: (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce] Jul 20 13:52:26 osd1 ceph-osd[4314]: 4: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x5565a245c5d8] Jul 20 13:52:26 osd1 ceph-osd[4314]: 5: (AsyncConnection::process()+0x1d4e) [0x5565a2678bde] Jul 20 13:52:26 osd1 ceph-osd[4314]: 6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148] Jul 20 13:52:26 osd1 ceph-osd[4314]: 7: (()+0xb0d0d8) [0x5565a24c70d8] Jul 20 13:52:26 osd1 ceph-osd[4314]: 8: (()+0xb8c80) [0x7feabad80c80] Jul 20 13:52:26 osd1 ceph-osd[4314]: 9: (()+0x76ba) [0x7feabb46f6ba] Jul 20 13:52:26 osd1 ceph-osd[4314]: 10: (clone()+0x6d) [0x7feaba4e63dd] Jul 20 13:52:26 osd1 ceph-osd[4314]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:52:26 osd1 ceph-osd[4314]: *** Caught signal (Aborted) ** Jul 20 13:52:26 osd1 ceph-osd[4314]: in thread 7feab78f9700 thread_name:msgr-worker-1 Jul 20 13:52:26 osd1 ceph-osd[4314]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:52:26 osd1 ceph-osd[4314]: 1: (()+0xa257a4) [0x5565a23df7a4] Jul 20 13:52:26 osd1 ceph-osd[4314]: 2: (()+0x11390) [0x7feabb479390] Jul 20 13:52:26 osd1 ceph-osd[4314]: 3: (gsignal()+0x38) [0x7feaba414428] Jul 20 13:52:26 osd1 ceph-osd[4314]: 4: (abort()+0x16a) [0x7feaba41602a] Jul 20 13:52:26 osd1 ceph-osd[4314]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5565a2421cfe] Jul 20 13:52:26 osd1 ceph-osd[4314]: 6: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9] Jul 20 13:52:26 osd1 ceph-osd[4314]: 7: (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce] Jul 20 13:52:26 osd1 ceph-osd[4314]: 8: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x5565a245c5d8] Jul 20 13:52:26 osd1 ceph-osd[4314]: 9: (AsyncConnection::process()+0x1d4e) [0x5565a2678bde] Jul 20 13:52:26 osd1 ceph-osd[4314]: 10: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148] Jul 20 13:52:26 osd1 ceph-osd[4314]: 11: (()+0xb0d0d8) [0x5565a24c70d8] Jul 20 13:52:26 osd1 ceph-osd[4314]: 12: (()+0xb8c80) [0x7feabad80c80] Jul 20 13:52:26 osd1 ceph-osd[4314]: 13: (()+0x76ba) [0x7feabb46f6ba] Jul 20 13:52:26 osd1 ceph-osd[4314]: 14: (clone()+0x6d) [0x7feaba4e63dd] Jul 20 13:52:26 osd1 ceph-osd[4314]: 2017-07-20 13:52:26.554188 7feab78f9700 -1 *** Caught signal (Aborted) ** Jul 20 13:52:26 osd1 ceph-osd[4314]: in thread 7feab78f9700 thread_name:msgr-worker-1 Jul 20 13:52:26 osd1 ceph-osd[4314]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:52:26 osd1 ceph-osd[4314]: 1: (()+0xa257a4) [0x5565a23df7a4] Jul 20 13:52:26 osd1 ceph-osd[4314]: 2: (()+0x11390) [0x7feabb479390] Jul 20 13:52:26 osd1 ceph-osd[4314]: 3: (gsignal()+0x38) [0x7feaba414428] Jul 20 13:52:26 osd1 ceph-osd[4314]: 4: (abort()+0x16a) [0x7feaba41602a] Jul 20 13:52:26 osd1 ceph-osd[4314]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5565a2421cfe] Jul 20 13:52:26 osd1 ceph-osd[4314]: 6: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9] Jul 20 13:52:26 osd1 ceph-osd[4314]: 7: (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce] Jul 20 13:52:26 osd1 ceph-osd[4314]: 8: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x5565a245c5d8] Jul 20 13:52:26 osd1 ceph-osd[4314]: 9: (AsyncConnection::process()+0x1d4e) [0x5565a2678bde] Jul 20 13:52:26 osd1 ceph-osd[4314]: 10: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148] Jul 20 13:52:26 osd1 ceph-osd[4314]: 11: (()+0xb0d0d8) [0x5565a24c70d8] Jul 20 13:52:26 osd1 ceph-osd[4314]: 12: (()+0xb8c80) [0x7feabad80c80] Jul 20 13:52:26 osd1 ceph-osd[4314]: 13: (()+0x76ba) [0x7feabb46f6ba] Jul 20 13:52:26 osd1 ceph-osd[4314]: 14: (clone()+0x6d) [0x7feaba4e63dd] Jul 20 13:52:26 osd1 ceph-osd[4314]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:52:26 osd1 ceph-osd[4314]: 0> 2017-07-20 13:52:26.554188 7feab78f9700 -1 *** Caught signal (Aborted) ** Jul 20 13:52:26 osd1 ceph-osd[4314]: in thread 7feab78f9700 thread_name:msgr-worker-1 Jul 20 13:52:26 osd1 ceph-osd[4314]: ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc) Jul 20 13:52:26 osd1 ceph-osd[4314]: 1: (()+0xa257a4) [0x5565a23df7a4] Jul 20 13:52:26 osd1 ceph-osd[4314]: 2: (()+0x11390) [0x7feabb479390] Jul 20 13:52:26 osd1 ceph-osd[4314]: 3: (gsignal()+0x38) [0x7feaba414428] Jul 20 13:52:26 osd1 ceph-osd[4314]: 4: (abort()+0x16a) [0x7feaba41602a] Jul 20 13:52:26 osd1 ceph-osd[4314]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5565a2421cfe] Jul 20 13:52:26 osd1 ceph-osd[4314]: 6: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9] Jul 20 13:52:26 osd1 ceph-osd[4314]: 7: (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce] Jul 20 13:52:26 osd1 ceph-osd[4314]: 8: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x5565a245c5d8] Jul 20 13:52:26 osd1 ceph-osd[4314]: 9: (AsyncConnection::process()+0x1d4e) [0x5565a2678bde] Jul 20 13:52:26 osd1 ceph-osd[4314]: 10: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148] Jul 20 13:52:26 osd1 ceph-osd[4314]: 11: (()+0xb0d0d8) [0x5565a24c70d8] Jul 20 13:52:26 osd1 ceph-osd[4314]: 12: (()+0xb8c80) [0x7feabad80c80] Jul 20 13:52:26 osd1 ceph-osd[4314]: 13: (()+0x76ba) [0x7feabb46f6ba] Jul 20 13:52:26 osd1 ceph-osd[4314]: 14: (clone()+0x6d) [0x7feaba4e63dd] Jul 20 13:52:26 osd1 ceph-osd[4314]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 20 13:52:26 osd1 systemd[1]: ceph-osd@3.service: Main process exited, code=killed, status=6/ABRT Jul 20 13:52:26 osd1 systemd[1]: ceph-osd@3.service: Unit entered failed state. Jul 20 13:52:26 osd1 systemd[1]: ceph-osd@3.service: Failed with result 'signal'. Jul 20 13:52:46 osd1 systemd[1]: ceph-osd@3.service: Service hold-off time over, scheduling restart. Jul 20 13:52:46 osd1 systemd[1]: Stopped Ceph object storage daemon osd.3. Jul 20 13:52:46 osd1 systemd[1]: Starting Ceph object storage daemon osd.3... Jul 20 13:52:47 osd1 systemd[1]: Started Ceph object storage daemon osd.3. Jul 20 13:52:47 osd1 ceph-osd[4406]: starting osd.3 at - osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com