I applied latest os updates and rebooted my hosts. Now all my osds fail to
start.
# cat /etc/os-release
NAME="openSUSE Tumbleweed"
# VERSION="20220207"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20220207"
# uname -a
Linux cube 5.16.5-1-default #1 SMP PREEMPT Thu Feb 3 05:26:48 UTC 2022
(1af4009) x86_64 x86_64 x86_64 GNU/Linux
container image: v16.2.7 / v16.2.7-20220201
osd debug log shows the following
-11> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 bluefs add_block_device
bdev 0 path /var/lib/ceph/osd/ceph-0/block.wal size 50 GiB
-10> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb option
max_total_wal_size = 1073741824
-9> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb option
compaction_readahead_size = 2097152
-8> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb option
max_write_buffer_number = 4
-7> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb option
max_background_compactions = 2
-6> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb option
compression = kNoCompression
-5> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb option
writable_file_max_buffer_size = 0
-4> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb option
min_write_buffer_number_to_merge = 1
-3> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb option
recycle_log_file_num = 4
-2> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb option
write_buffer_size = 268435456
-1> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 bluefs mount
0> 2022-02-10T19:14:48.387-0800 7ff1be4c3080 -1 *** Caught signal
(Aborted) **
in thread 7ff1be4c3080 thread_name:ceph-osd
ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific
(stable)
1: /lib64/libpthread.so.0(+0x12c20) [0x7ff1bc465c20]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff1bba7c09b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7ff1bba8253c]
6: /lib64/libstdc++.so.6(+0x96597) [0x7ff1bba82597]
7: /lib64/libstdc++.so.6(+0x967f8) [0x7ff1bba827f8]
8: ceph-osd(+0x56301f) [0x559ff6d6301f]
9: (BlueFS::_open_super()+0x18c) [0x559ff745f08c]
10: (BlueFS::mount()+0xeb) [0x559ff748085b]
11: (BlueStore::_open_bluefs(bool, bool)+0x94) [0x559ff735e464]
12: (BlueStore::_prepare_db_environment(bool, bool,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >*, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >*)+0x6d9) [0x559ff735f5b9]
13: (BlueStore::_open_db(bool, bool, bool)+0x155) [0x559ff73608b5]
14: (BlueStore::_open_db_and_around(bool, bool)+0x273) [0x559ff73cba33]
15: (BlueStore::_mount()+0x204) [0x559ff73ce974]
16: (OSD::init()+0x380) [0x559ff6ea2400]
17: main()
18: __libc_start_main()
19: _start()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
to interpret this.
The process log shows the following
2022-02-10T19:33:31.852-0800 7f22869e8080 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore,rocksdb
2022-02-10T19:33:31.852-0800 7f22869e8080 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore,rocksdb
2022-02-10T19:33:31.852-0800 7f22869e8080 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore,rocksdb
terminate called after throwing an instance of
'ceph::buffer::v15_2_0::malformed_input'
what(): void
bluefs_super_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no
longer understand old encoding version 2 < 143: Malformed input
*** Caught signal (Aborted) **
in thread 7f22869e8080 thread_name:ceph-osd
ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific
(stable)
1: /lib64/libpthread.so.0(+0x12c20) [0x7f228498ac20]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7f2283fa109b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7f2283fa753c]
6: /lib64/libstdc++.so.6(+0x96597) [0x7f2283fa7597]
7: /lib64/libstdc++.so.6(+0x967f8) [0x7f2283fa77f8]
8: ceph-osd(+0x56301f) [0x55e6faf6301f]
9: (BlueFS::_open_super()+0x18c) [0x55e6fb65f08c]
10: (BlueFS::mount()+0xeb) [0x55e6fb68085b]
11: (BlueStore::_open_bluefs(bool, bool)+0x94) [0x55e6fb55e464]
12: (BlueStore::_prepare_db_environment(bool, bool,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >*, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >*)+0x6d9) [0x55e6fb55f5b9]
13: (BlueStore::_open_db(bool, bool, bool)+0x155) [0x55e6fb5608b5]
14: (BlueStore::_open_db_and_around(bool, bool)+0x273) [0x55e6fb5cba33]
15: (BlueStore::_mount()+0x204) [0x55e6fb5ce974]
16: (OSD::init()+0x380) [0x55e6fb0a2400]
17: main()
18: __libc_start_main()
19: _start()
2022-02-10T19:33:34.620-0800 7f22869e8080 -1 *** Caught signal (Aborted) **
in thread 7f22869e8080 thread_name:ceph-osd
Doesn't anyone have any ideas what could be going on here?
Thanks,
/Chris
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
