I have an issue with incomplete pgs, I've tried repairing it but no such luck.
Any ideas what to check?
Output from 'ceph health detail'
HEALTH_ERR 2 pgs inconsistent; 1 pgs recovering; 1 pgs stuck unclean; recovery
15/863113 degraded (0.002%); 5/287707 unfound (0.002%); 4 scrub errors
pg 22.ee is stuck unclean for 131473.768406, current state
active+recovering+inconsistent, last acting [45,16,21]
pg 22.ee is active+recovering+inconsistent, acting [45,16,21], 5 unfound
pg 22.4a is active+clean+inconsistent, acting [2,25,34]
recovery 15/863113 degraded (0.002%); 5/287707 unfound (0.002%)
4 scrub errors
I tried to remove one of the nodes and now the service crashes on startup
Dec 12 22:56:32 ceph12 ceph-osd: 0> 2013-12-12 22:56:32.000946
7fe4dcd4a700 -1 *** Caught signal (Aborted) **#012 in thread
7fe4dcd4a700#012#012 ceph version 0.67.4
(ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)#012 1: /usr/bin/ceph-osd()
[0x8001ea]#012 2: (()+0xfcb0) [0x7fe4f029bcb0]#012 3: (gsignal()+0x35)
[0x7fe4eea53425]#012 4: (abort()+0x17b) [0x7fe4eea56b8b]#012 5:
(__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fe4ef3a669d]#012 6:
(()+0xb5846) [0x7fe4ef3a4846]#012 7: (()+0xb5873) [0x7fe4ef3a4873]#012 8:
(()+0xb596e) [0x7fe4ef3a496e]#012 9:
(ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x8c7087]#012
10: (object_info_t::decode(ceph::buffer::list::iterator&)+0x73) [0x95c163]#012
11: (ReplicatedPG::build_push_op(ObjectRecoveryInfo const&,
ObjectRecoveryProgress const&, ObjectRecoveryProgress*, PushOp*)+0x87f)
[0x5f123f]#012 12: (ReplicatedPG::handle_pull(int, PullOp&, PushOp*)+0xc1)
[0x5f4611]#012 13:
(ReplicatedPG::do_pull(std::tr1::shared_ptr<OpRequest>)+0x4f4) [0x5f53b4]#012
14: (PG::do_request(std::tr1::shared_ptr<OpRequest>,
ThreadPool::TPHandle&)+0x348) [0x703e38]#012 15:
(OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>,
ThreadPool::TPHandle&)+0x330) [0x658620]#012 16:
(OSD::OpWQ::_process(boost::intrusive_ptr<PG>, ThreadPool::TPHandle&)+0x4a0)
[0x66ed10]#012 17:
(ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>,
std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG>
>::_void_process(void*, ThreadPool::TPHandle&)+0x9c) [0x6aa25c]#012 18:
(ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x8b8f96]#012 19:
(ThreadPool::WorkThread::entry()+0x10) [0x8bada0]#012 20: (()+0x7e9a)
[0x7fe4f0293e9a]#012 21: (clone()+0x6d) [0x7fe4eeb113fd]#012 NOTE: a copy of
the executable, or `objdump -rdS <executable>` is needed to interpret this.
Nelson Jeppesen____________
Disney Technology Solutions and Services
Phone 206-588-5001
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
