Hello, Can anyone help me interpret the below error, which one of our OSDs has been occasionally throwing since last night. Thanks.
----- Jan 29 03:00:53 osd-host ceph-osd[10964]: 2018-01-29 03:00:53.509185 7fe4ae431700 -1 bluestore(/var/lib/ceph/osd/ceph-9) _balance_bluefs_freespace allocate failed on 0x80000000 min_alloc_size 0x4000 Jan 29 03:00:57 osd-host ceph-osd[10964]: /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_balance_bluefs_freespace(PExtentVector*)' thread 7fe4ae431700 time 2018-01-29 03:00:57.736207 Jan 29 03:00:57 osd-host ceph-osd[10964]: /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 4939: FAILED assert(0 == "allocate failed, wtf") Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55e88d2ae892] Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >*)+0x1b21) [0x55e88d1405c1] Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040] Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d] Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (()+0x76da) [0x7fe4bec766da] Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (clone()+0x5f) [0x7fe4bdce8d7f] Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jan 29 03:00:57 osd-host ceph-osd[10964]: 2018-01-29 03:00:57.741058 7fe4ae431700 -1 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_balance_bluefs_freespace(PExtentVector*)' thread 7fe4ae431700 time 201 8-01-29 03:00:57.736207 Jan 29 03:00:57 osd-host ceph-osd[10964]: /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 4939: FAILED assert(0 == "allocate failed, wtf") Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55e88d2ae892] Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >*)+0x1b21) [0x55e88d1405c1] Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040] Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d] Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (()+0x76da) [0x7fe4bec766da] Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (clone()+0x5f) [0x7fe4bdce8d7f] Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jan 29 03:00:57 osd-host ceph-osd[10964]: 0> 2018-01-29 03:00:57.741058 7fe4ae431700 -1 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_balance_bluefs_freespace(PExtentVector*)' thread 7fe4ae431700 time 2018-01-29 03:00:57.736207 Jan 29 03:00:57 osd-host ceph-osd[10964]: /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 4939: FAILED assert(0 == "allocate failed, wtf") Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55e88d2ae892] Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >*)+0x1b21) [0x55e88d1405c1] Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040] Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d] Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (()+0x76da) [0x7fe4bec766da] Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (clone()+0x5f) [0x7fe4bdce8d7f] Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jan 29 03:00:57 osd-host ceph-osd[10964]: *** Caught signal (Aborted) ** Jan 29 03:00:57 osd-host ceph-osd[10964]: in thread 7fe4ae431700 thread_name:bstore_kv_sync Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (()+0xa65824) [0x55e88d26b824] Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (()+0x11670) [0x7fe4bec80670] Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (gsignal()+0x9f) [0x7fe4bdc1577f] Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (abort()+0x16a) [0x7fe4bdc1737a] Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55e88d2aea1e] Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >*)+0x1b21) [0x55e88d1405c1] Jan 29 03:00:57 osd-host ceph-osd[10964]: 7: (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040] Jan 29 03:00:57 osd-host ceph-osd[10964]: 8: (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d] Jan 29 03:00:57 osd-host ceph-osd[10964]: 9: (()+0x76da) [0x7fe4bec766da] Jan 29 03:00:57 osd-host ceph-osd[10964]: 10: (clone()+0x5f) [0x7fe4bdce8d7f] Jan 29 03:00:57 osd-host ceph-osd[10964]: 2018-01-29 03:00:57.790980 7fe4ae431700 -1 *** Caught signal (Aborted) ** Jan 29 03:00:57 osd-host ceph-osd[10964]: in thread 7fe4ae431700 thread_name:bstore_kv_sync Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (()+0xa65824) [0x55e88d26b824] Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (()+0x11670) [0x7fe4bec80670] Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (gsignal()+0x9f) [0x7fe4bdc1577f] Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (abort()+0x16a) [0x7fe4bdc1737a] Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55e88d2aea1e] Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >*)+0x1b21) [0x55e88d1405c1] Jan 29 03:00:57 osd-host ceph-osd[10964]: 7: (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040] Jan 29 03:00:57 osd-host ceph-osd[10964]: 8: (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d] Jan 29 03:00:57 osd-host ceph-osd[10964]: 9: (()+0x76da) [0x7fe4bec766da] Jan 29 03:00:57 osd-host ceph-osd[10964]: 10: (clone()+0x5f) [0x7fe4bdce8d7f] Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jan 29 03:00:57 osd-host ceph-osd[10964]: 0> 2018-01-29 03:00:57.790980 7fe4ae431700 -1 *** Caught signal (Aborted) ** Jan 29 03:00:57 osd-host ceph-osd[10964]: in thread 7fe4ae431700 thread_name:bstore_kv_sync Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (()+0xa65824) [0x55e88d26b824] Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (()+0x11670) [0x7fe4bec80670] Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (gsignal()+0x9f) [0x7fe4bdc1577f] Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (abort()+0x16a) [0x7fe4bdc1737a] Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55e88d2aea1e] Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >*)+0x1b21) [0x55e88d1405c1] Jan 29 03:00:57 osd-host ceph-osd[10964]: 7: (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040] Jan 29 03:00:57 osd-host ceph-osd[10964]: 8: (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d] Jan 29 03:00:57 osd-host ceph-osd[10964]: 9: (()+0x76da) [0x7fe4bec766da] Jan 29 03:00:57 osd-host ceph-osd[10964]: 10: (clone()+0x5f) [0x7fe4bdce8d7f] Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jan 29 03:00:58 osd-host systemd[1]: [email protected]: Main process exited, code=killed, status=6/ABRT -----
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
