Hi,
One of several OSDs on the same machine crashed several times within days.
It's always that one, other OSDs are all fine. Below is the dumped message,
since it's too long here, I only pasted the head and tail of the recent
events. If it's necessary to inspect the full log, please see
https://gist.github.com/dotSlashLu/3e8ca9491fbf07636a4583244ac23f80.
2016-10-24 18:52:06.216341 7f307c22f700 -1 os/FileStore.cc: In function
'virtual int FileStore::read(coll_t, const ghobject_t&, uint64_t, size_t,
ceph::bufferlist&, uint32_t, bool)' thread 7f307c22f700 time 2016-10-24
18:52:06.213123
os/FileStore.cc: 2854: FAILED assert(allow_eio || !m_filestore_fail_eio ||
got != -5)
ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x85) [0xbc9195]
2: (FileStore::read(coll_t, ghobject_t const&, unsigned long, unsigned
long, ceph::buffer::list&, unsigned int, bool)+0xc94) [0x909f34]
3: (ReplicatedBackend::be_deep_scrub(hobject_t const&, unsigned int,
ScrubMap::object&, ThreadPool::TPHandle&)+0x311) [0x9fe0e1]
4: (PGBackend::be_scan_list(ScrubMap&, std::vector<hobject_t,
std::allocator<hobject_t> > const&, bool, unsigned int,
ThreadPool::TPHandle&)+0x2e8) [0x8ce8c8]
5: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, bool,
unsigned int, ThreadPool::TPHandle&)+0x213) [0x7def53]
6: (PG::replica_scrub(MOSDRepScrub*, ThreadPool::TPHandle&)+0x4c2)
[0x7df722]
7: (OSD::RepScrubWQ::_process(MOSDRepScrub*, ThreadPool::TPHandle&)+0xbe)
[0x6dcade]
8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa76) [0xbb9966]
9: (ThreadPool::WorkThread::entry()+0x10) [0xbba9f0]
10: (()+0x7dc5) [0x7f309cd26dc5]
11: (clone()+0x6d) [0x7f309b80821d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
to interpret this.
--- begin dump of recent events ---
-10000> 2016-10-24 18:51:34.341035 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.56:6821/4808 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x175a2c00 con 0x1526a940
-9999> 2016-10-24 18:51:34.341046 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.61:6817/4808 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x175a3600 con 0x15269fa0
-9998> 2016-10-24 18:51:34.341058 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.56:6823/5402 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x12aaa400 con 0x27bc9080
-9997> 2016-10-24 18:51:34.341069 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.61:6821/5402 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x1f89ec00 con 0x27bc91e0
-9996> 2016-10-24 18:51:34.341080 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.56:6824/6216 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0xaa16000 con 0x175b0c00
-9995> 2016-10-24 18:51:34.341090 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.61:6818/6216 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x23b87800 con 0x175ae160
-9994> 2016-10-24 18:51:34.341101 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.57:6802/23367 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x258ed400 con 0x17500d60
-9993> 2016-10-24 18:51:34.341113 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.62:6806/23367 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x242bb000 con 0x175019c0
-9992> 2016-10-24 18:51:34.341128 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.57:6805/25009 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x28e41c00 con 0x1744aec0
-9991> 2016-10-24 18:51:34.341139 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.62:6805/25009 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x10be5200 con 0x175bf8c0
-9990> 2016-10-24 18:51:34.341130 7f3088a48700 1 -- 10.3.149.62:0/25857
<== osd.1 10.3.149.55:6835/2010188 187557 ==== osd_ping(ping_reply e3014
stamp 2016-10-24 18:51:34.340550) v2 ==== 47+0+0 (1550182756 0 0)
0x1a83bc00 con 0x7874580
-9989> 2016-10-24 18:51:34.341151 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.57:6814/26469 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x1f48aa00 con 0x175bfa20
-9988> 2016-10-24 18:51:34.341162 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.62:6811/26469 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x24456e00 con 0x175bfb80
-9987> 2016-10-24 18:51:34.341174 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.58:6805/2023199 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x25c59e00 con 0x7874f20
-9986> 2016-10-24 18:51:34.341186 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.63:6805/2023199 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x19703c00 con 0x7875760
-9985> 2016-10-24 18:51:34.341208 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.58:6803/2023356 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x19702600 con 0x26444940
-9984> 2016-10-24 18:51:34.341231 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.63:6803/2023356 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0xa67da00 con 0x7874c60
-9983> 2016-10-24 18:51:34.341249 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.58:6809/2023604 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x22111000 con 0x17887860
-9982> 2016-10-24 18:51:34.341262 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.63:6811/2023604 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x1fe62200 con 0x17887de0
-9981> 2016-10-24 18:51:34.341281 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.58:6802/2023892 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x1fc32c00 con 0x24246100
-9980> 2016-10-24 18:51:34.341297 7f307b22d700 1 -- 10.3.149.62:0/25857
--> 10.3.149.63:6801/2023892 -- osd_ping(ping e3014 stamp 2016-10-24
18:51:34.340550) v2 -- ?+0 0x20544c00 con 0x24246d60
.
.
.
-20> 2016-10-24 18:52:05.273121 7f3086243700 1 -- 10.3.149.57:6811/25857
--> 10.3.149.60:0/10188 -- osd_ping(ping_reply e3014 stamp 2016-10-24
18:52:05.212809) v2 -- ?+0 0x27c1a600 con 0x1744aaa0
-19> 2016-10-24 18:52:05.273129 7f3087a46700 1 -- 10.3.149.62:6810/25857
<== osd.1 10.3.149.60:0/10188 187279 ==== osd_ping(ping e3014 stamp
2016-10-24 18:52:05.212809) v2 ==== 47+0+0 (387409057 0 0) 0x1ff4f600 con
0x175b1860
-18> 2016-10-24 18:52:05.273157 7f3087a46700 1 -- 10.3.149.62:6810/25857
--> 10.3.149.60:0/10188 -- osd_ping(ping_reply e3014 stamp 2016-10-24
18:52:05.212809) v2 -- ?+0 0x10d73a00 con 0x175b1860
-17> 2016-10-24 18:52:05.641202 7f3086243700 1 -- 10.3.149.57:6811/25857
<== osd.29 10.3.149.59:0/35501 187818 ==== osd_ping(ping e3014 stamp
2016-10-24 18:52:05.640915) v2 ==== 47+0+0 (3027252596 0 0) 0x9d0a200 con
0x175172e0
-16> 2016-10-24 18:52:05.641209 7f3087a46700 1 -- 10.3.149.62:6810/25857
<== osd.29 10.3.149.59:0/35501 187818 ==== osd_ping(ping e3014 stamp
2016-10-24 18:52:05.640915) v2 ==== 47+0+0 (3027252596 0 0) 0xa27ba00 con
0x264422c0
-15> 2016-10-24 18:52:05.641246 7f3086243700 1 -- 10.3.149.57:6811/25857
--> 10.3.149.59:0/35501 -- osd_ping(ping_reply e3014 stamp 2016-10-24
18:52:05.640915) v2 -- ?+0 0x1b8a6200 con 0x175172e0
-14> 2016-10-24 18:52:05.641290 7f3087a46700 1 -- 10.3.149.62:6810/25857
--> 10.3.149.59:0/35501 -- osd_ping(ping_reply e3014 stamp 2016-10-24
18:52:05.640915) v2 -- ?+0 0x1ff4f600 con 0x264422c0
-13> 2016-10-24 18:52:05.689610 7f3086243700 1 -- 10.3.149.57:6811/25857
<== osd.13 10.3.149.56:0/5402 187624 ==== osd_ping(ping e3014 stamp
2016-10-24 18:52:05.634215) v2 ==== 47+0+0 (1310408758 0 0) 0x1be24600 con
0x15268b00
-12> 2016-10-24 18:52:05.689664 7f3086243700 1 -- 10.3.149.57:6811/25857
--> 10.3.149.56:0/5402 -- osd_ping(ping_reply e3014 stamp 2016-10-24
18:52:05.634215) v2 -- ?+0 0x9d0a200 con 0x15268b00
-11> 2016-10-24 18:52:05.689661 7f3087a46700 1 -- 10.3.149.62:6810/25857
<== osd.13 10.3.149.56:0/5402 187624 ==== osd_ping(ping e3014 stamp
2016-10-24 18:52:05.634215) v2 ==== 47+0+0 (1310408758 0 0) 0x19705600 con
0x175b1de0
-10> 2016-10-24 18:52:05.689729 7f3087a46700 1 -- 10.3.149.62:6810/25857
--> 10.3.149.56:0/5402 -- osd_ping(ping_reply e3014 stamp 2016-10-24
18:52:05.634215) v2 -- ?+0 0xa27ba00 con 0x175b1de0
-9> 2016-10-24 18:52:05.861925 7f3086243700 1 -- 10.3.149.57:6811/25857
<== osd.4 10.3.149.60:0/12742 187653 ==== osd_ping(ping e3014 stamp
2016-10-24 18:52:05.801655) v2 ==== 47+0+0 (350590821 0 0) 0x12169400 con
0x17514000
-8> 2016-10-24 18:52:05.861957 7f3086243700 1 -- 10.3.149.57:6811/25857
--> 10.3.149.60:0/12742 -- osd_ping(ping_reply e3014 stamp 2016-10-24
18:52:05.801655) v2 -- ?+0 0x1be24600 con 0x17514000
-7> 2016-10-24 18:52:05.861963 7f3087a46700 1 -- 10.3.149.62:6810/25857
<== osd.4 10.3.149.60:0/12742 187653 ==== osd_ping(ping e3014 stamp
2016-10-24 18:52:05.801655) v2 ==== 47+0+0 (350590821 0 0) 0x269fba00 con
0x26442840
-6> 2016-10-24 18:52:05.862015 7f3087a46700 1 -- 10.3.149.62:6810/25857
--> 10.3.149.60:0/12742 -- osd_ping(ping_reply e3014 stamp 2016-10-24
18:52:05.801655) v2 -- ?+0 0x19705600 con 0x26442840
-5> 2016-10-24 18:52:05.882605 7f3094bb6700 5 osd.19 3014 tick
-4> 2016-10-24 18:52:05.988572 7f3086243700 1 -- 10.3.149.57:6811/25857
<== osd.25 10.3.149.58:0/24382 187898 ==== osd_ping(ping e3014 stamp
2016-10-24 18:52:05.984426) v2 ==== 47+0+0 (3778423740 0 0) 0xae91200 con
0x177bb760
-3> 2016-10-24 18:52:05.988582 7f3087a46700 1 -- 10.3.149.62:6810/25857
<== osd.25 10.3.149.58:0/24382 187898 ==== osd_ping(ping e3014 stamp
2016-10-24 18:52:05.984426) v2 ==== 47+0+0 (3778423740 0 0) 0x1a396000 con
0x1526bc80
-2> 2016-10-24 18:52:05.988608 7f3086243700 1 -- 10.3.149.57:6811/25857
--> 10.3.149.58:0/24382 -- osd_ping(ping_reply e3014 stamp 2016-10-24
18:52:05.984426) v2 -- ?+0 0x12169400 con 0x177bb760
-1> 2016-10-24 18:52:05.988652 7f3087a46700 1 -- 10.3.149.62:6810/25857
--> 10.3.149.58:0/24382 -- osd_ping(ping_reply e3014 stamp 2016-10-24
18:52:05.984426) v2 -- ?+0 0x269fba00 con 0x1526bc80
0> 2016-10-24 18:52:06.216341 7f307c22f700 -1 os/FileStore.cc: In
function 'virtual int FileStore::read(coll_t, const ghobject_t&, uint64_t,
size_t, ceph::bufferlist&, uint32_t, bool)' thread 7f307c22f700 time
2016-10-24 18:52:06.213123
os/FileStore.cc: 2854: FAILED assert(allow_eio || !m_filestore_fail_eio ||
got != -5)
ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x85) [0xbc9195]
2: (FileStore::read(coll_t, ghobject_t const&, unsigned long, unsigned
long, ceph::buffer::list&, unsigned int, bool)+0xc94) [0x909f34]
3: (ReplicatedBackend::be_deep_scrub(hobject_t const&, unsigned int,
ScrubMap::object&, ThreadPool::TPHandle&)+0x311) [0x9fe0e1]
4: (PGBackend::be_scan_list(ScrubMap&, std::vector<hobject_t,
std::allocator<hobject_t> > const&, bool, unsigned int,
ThreadPool::TPHandle&)+0x2e8) [0x8ce8c8]
5: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, bool,
unsigned int, ThreadPool::TPHandle&)+0x213) [0x7def53]
6: (PG::replica_scrub(MOSDRepScrub*, ThreadPool::TPHandle&)+0x4c2)
[0x7df722]
7: (OSD::RepScrubWQ::_process(MOSDRepScrub*, ThreadPool::TPHandle&)+0xbe)
[0x6dcade]
8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa76) [0xbb9966]
9: (ThreadPool::WorkThread::entry()+0x10) [0xbba9f0]
10: (()+0x7dc5) [0x7f309cd26dc5]
11: (clone()+0x6d) [0x7f309b80821d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.19.log
--- end dump of recent events ---
Since ceph-osd objdump is too large to put in a mail, I will not attach it,
but if it is needed i'll find a way to share it. What might be the cause?
Can any one help me with this? Thanks.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
