Hi,
Another OSD falled down. And it's pretty scary how easy is to break the
cluster. This time is something related to the journal.
/usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
starting osd.6 at :/0 osd_data /var/lib/ceph/osd/ceph-6
/var/lib/ceph/osd/ceph-6/journal
2017-12-05 13:19:03.473082 7f24515148c0 -1 osd.6 10538 log_to_monitors
{default=true}
os/filestore/FileStore.cc: In function 'void
FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
ThreadPool::TPHandle*)' thread 7f243d1a0700 time 2017-12-05 13:19:04.433036
os/filestore/FileStore.cc: 2930: FAILED assert(0 == "unexpected error")
ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x55569c1ff790]
2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
long, int, ThreadPool::TPHandle*)+0xb8e) [0x55569be9d58e]
3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction,
std::allocator<ObjectStore::Transaction> >&, unsigned long,
ThreadPool::TPHandle*)+0x3b) [0x55569bea3a1b]
4: (FileStore::_do_op(FileStore::OpSequencer*,
ThreadPool::TPHandle&)+0x39d) [0x55569bea3ded]
5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb1) [0x55569c1f1961]
6: (ThreadPool::WorkThread::entry()+0x10) [0x55569c1f2a60]
7: (()+0x76ba) [0x7f24503e36ba]
8: (clone()+0x6d) [0x7f244e45b3dd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
2017-12-05 13:19:04.437968 7f243d1a0700 -1 os/filestore/FileStore.cc: In
function 'void FileStore::_do_transaction(ObjectStore::Transaction&,
uint64_t, int, ThreadPool::TPHandle*)' thread 7f243d1a0700 time
2017-12-05 13:19:04.433036
os/filestore/FileStore.cc: 2930: FAILED assert(0 == "unexpected error")
ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x55569c1ff790]
2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
long, int, ThreadPool::TPHandle*)+0xb8e) [0x55569be9d58e]
3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction,
std::allocator<ObjectStore::Transaction> >&, unsigned long,
ThreadPool::TPHandle*)+0x3b) [0x55569bea3a1b]
4: (FileStore::_do_op(FileStore::OpSequencer*,
ThreadPool::TPHandle&)+0x39d) [0x55569bea3ded]
5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb1) [0x55569c1f1961]
6: (ThreadPool::WorkThread::entry()+0x10) [0x55569c1f2a60]
7: (()+0x76ba) [0x7f24503e36ba]
8: (clone()+0x6d) [0x7f244e45b3dd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
os/filestore/FileStore.cc: In function 'void
FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
ThreadPool::TPHandle*)' thread 7f243d9a1700 time 2017-12-05 13:19:04.435362
os/filestore/FileStore.cc: 2930: FAILED assert(0 == "unexpected error")
ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x55569c1ff790]
2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
long, int, ThreadPool::TPHandle*)+0xb8e) [0x55569be9d58e]
3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction,
std::allocator<ObjectStore::Transaction> >&, unsigned long,
ThreadPool::TPHandle*)+0x3b) [0x55569bea3a1b]
4: (FileStore::_do_op(FileStore::OpSequencer*,
ThreadPool::TPHandle&)+0x39d) [0x55569bea3ded]
5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb1) [0x55569c1f1961]
6: (ThreadPool::WorkThread::entry()+0x10) [0x55569c1f2a60]
7: (()+0x76ba) [0x7f24503e36ba]
8: (clone()+0x6d) [0x7f244e45b3dd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
-405> 2017-12-05 13:19:03.473082 7f24515148c0 -1 osd.6 10538
log_to_monitors {default=true}
0> 2017-12-05 13:19:04.437968 7f243d1a0700 -1
os/filestore/FileStore.cc: In function 'void
FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
ThreadPool::TPHandle*)' thread 7f243d1a0700 time 2017-12-05 13:19:04.433036
os/filestore/FileStore.cc: 2930: FAILED assert(0 == "unexpected error")
ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x55569c1ff790]
2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
long, int, ThreadPool::TPHandle*)+0xb8e) [0x55569be9d58e]
3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction,
std::allocator<ObjectStore::Transaction> >&, unsigned long,
ThreadPool::TPHandle*)+0x3b) [0x55569bea3a1b]
4: (FileStore::_do_op(FileStore::OpSequencer*,
ThreadPool::TPHandle&)+0x39d) [0x55569bea3ded]
5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb1) [0x55569c1f1961]
6: (ThreadPool::WorkThread::entry()+0x10) [0x55569c1f2a60]
7: (()+0x76ba) [0x7f24503e36ba]
8: (clone()+0x6d) [0x7f244e45b3dd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
2017-12-05 13:19:04.442866 7f243d9a1700 -1 os/filestore/FileStore.cc: In
function 'void FileStore::_do_transaction(ObjectStore::Transaction&,
uint64_t, int, ThreadPool::TPHandle*)' thread 7f243d9a1700 time
2017-12-05 13:19:04.435362
os/filestore/FileStore.cc: 2930: FAILED assert(0 == "unexpected error")
ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x55569c1ff790]
2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
long, int, ThreadPool::TPHandle*)+0xb8e) [0x55569be9d58e]
3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction,
std::allocator<ObjectStore::Transaction> >&, unsigned long,
ThreadPool::TPHandle*)+0x3b) [0x55569bea3a1b]
4: (FileStore::_do_op(FileStore::OpSequencer*,
ThreadPool::TPHandle&)+0x39d) [0x55569bea3ded]
5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb1) [0x55569c1f1961]
6: (ThreadPool::WorkThread::entry()+0x10) [0x55569c1f2a60]
7: (()+0x76ba) [0x7f24503e36ba]
8: (clone()+0x6d) [0x7f244e45b3dd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
0> 2017-12-05 13:19:04.442866 7f243d9a1700 -1
os/filestore/FileStore.cc: In function 'void
FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
ThreadPool::TPHandle*)' thread 7f243d9a1700 time 2017-12-05 13:19:04.435362
os/filestore/FileStore.cc: 2930: FAILED assert(0 == "unexpected error")
ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x55569c1ff790]
2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
long, int, ThreadPool::TPHandle*)+0xb8e) [0x55569be9d58e]
3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction,
std::allocator<ObjectStore::Transaction> >&, unsigned long,
ThreadPool::TPHandle*)+0x3b) [0x55569bea3a1b]
4: (FileStore::_do_op(FileStore::OpSequencer*,
ThreadPool::TPHandle&)+0x39d) [0x55569bea3ded]
5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb1) [0x55569c1f1961]
6: (ThreadPool::WorkThread::entry()+0x10) [0x55569c1f2a60]
7: (()+0x76ba) [0x7f24503e36ba]
8: (clone()+0x6d) [0x7f244e45b3dd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
*** Caught signal (Aborted) **
in thread 7f243d1a0700 thread_name:tp_fstore_op
I tried to boot it several times.
I zero the journal
dd if=/dev/zero of=/dev/sde2
create a new journal
ceph-osd --mkjournal -i 6
Flush it. But's empty so ok.
/usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup
ceph --flush-journal
and boot manually the osd.
/usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
Then it breaks. I pasted bin my whole configuration in
https://pastebin.com/QfrE71Dg.
But I changed also the journal partition from sde4 to sde2 to see if
this has something to do. sde is SSD disk so wanted to see no block is
corrupting everything.
Nothing it breaks 100% of time after a while. I'm desperate to see how
it breaks. I must say that this is other OSD that failed and I
recovered. Smartscan long is correct xfs_repair is ok on disk everything
seems correct. But it keep crashing.
Any advice?
Can I run the disk without journal for a while until all pg are backup
to the other disks? I just increased the size of the pools and min size
as well and I need this disk in order to recover all information.
Best regards,
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
