Hmm, I think this might actually be another instance of http://tracker.ceph.com/issues/8232, which was just reported yesterday. That said, I think that if you restart one OSD at a time, you should be able to avoid the race condition. It was restarting all of them simultaneously that got you into trouble. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com
On Tue, Apr 29, 2014 at 12:25 AM, Thanh Tran <[email protected]> wrote: > Hi, > > I upgraded my ceph from emperor to firefly (v0.80-rc1-16-g2708c3c), i > restart the whole ceph after finishing the upgrade. > After restarting, ceph begin to perform checking and recovering pgs, and > osds begin randomly to up and down constantly, some osds were crashed and > marked as down. I tried to start the crashed osds manually, but a later time > the other osds crashed. > > My cluster have 3 mons and 24 osds running on 3 hosts. > Ceph is upgraded from "deb > http://gitbuilder.ceph.com/ceph-deb-quantal-x86_64-basic/ref/firefly quantal > main". > > log information of one osd crashed: > -1> 2014-04-28 17:42:36.751933 7f6aacb30700 2 -- > 10.76.0.44:6814/192022485 >> 10.76.0.42:6800/5060598 pipe(0x2af92c80 sd=26 > :17458 s=1 pgs=0 cs=0 l=0 c=0x77636c60). got newly_acked_seq 226 vs out_seq > 0 > 0> 2014-04-28 17:42:36.752674 7f6ab293a700 -1 msg/Pipe.cc: In function > 'int Pipe::connect()' thread 7f6ab293a700 time 2014-04-28 17:42:36.750997 > msg/Pipe.cc: 1070: FAILED assert(m) > > ceph version 0.80-rc1-16-g2708c3c > (2708c3c559d99e6f3b557ee1d223efa3745f655c) > 1: (Pipe::connect()+0x3b61) [0xc18ab1] > 2: (Pipe::writer()+0x65f) [0xc193ef] > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] > 5: (clone()+0x6d) [0x7f6b089523fd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 keyvaluestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.19.log > --- end dump of recent events --- > 2014-04-28 17:42:36.756085 7f6aacb30700 -1 msg/Pipe.cc: In function 'int > Pipe::connect()' thread 7f6aacb30700 time 2014-04-28 17:42:36.751971 > msg/Pipe.cc: 1070: FAILED assert(m) > > ceph version 0.80-rc1-16-g2708c3c > (2708c3c559d99e6f3b557ee1d223efa3745f655c) > 1: (Pipe::connect()+0x3b61) [0xc18ab1] > 2: (Pipe::writer()+0x65f) [0xc193ef] > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] > 5: (clone()+0x6d) [0x7f6b089523fd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > 2014-04-28 17:42:36.760152 7f6ab7f54700 -1 msg/Pipe.cc: In function 'int > Pipe::connect()' thread 7f6ab7f54700 time 2014-04-28 17:42:36.753156 > msg/Pipe.cc: 1070: FAILED assert(m) > > ceph version 0.80-rc1-16-g2708c3c > (2708c3c559d99e6f3b557ee1d223efa3745f655c) > 1: (Pipe::connect()+0x3b61) [0xc18ab1] > 2: (Pipe::writer()+0x65f) [0xc193ef] > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] > 5: (clone()+0x6d) [0x7f6b089523fd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > 2014-04-28 17:42:36.765117 7f6ab9792700 -1 msg/Pipe.cc: In function 'int > Pipe::connect()' thread 7f6ab9792700 time 2014-04-28 17:42:36.763390 > msg/Pipe.cc: 1070: FAILED assert(m) > > ceph version 0.80-rc1-16-g2708c3c > (2708c3c559d99e6f3b557ee1d223efa3745f655c) > 1: (Pipe::connect()+0x3b61) [0xc18ab1] > 2: (Pipe::writer()+0x65f) [0xc193ef] > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] > 5: (clone()+0x6d) [0x7f6b089523fd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > --- begin dump of recent events --- > -4> 2014-04-28 17:42:36.753061 7f6ab7f54700 2 -- > 10.76.0.44:6814/192022485 >> 10.76.0.42:6819/3061991 pipe(0x771e2280 sd=115 > :47532 s=1 pgs=0 cs=0 l=0 c=0x3ea4f760). got newly_acked_seq 475 vs out_seq > 0 > -3> 2014-04-28 17:42:36.756085 7f6aacb30700 -1 msg/Pipe.cc: In function > 'int Pipe::connect()' thread 7f6aacb30700 time 2014-04-28 17:42:36.751971 > msg/Pipe.cc: 1070: FAILED assert(m) > > ceph version 0.80-rc1-16-g2708c3c > (2708c3c559d99e6f3b557ee1d223efa3745f655c) > 1: (Pipe::connect()+0x3b61) [0xc18ab1] > 2: (Pipe::writer()+0x65f) [0xc193ef] > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] > 5: (clone()+0x6d) [0x7f6b089523fd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > -2> 2014-04-28 17:42:36.760152 7f6ab7f54700 -1 msg/Pipe.cc: In function > 'int Pipe::connect()' thread 7f6ab7f54700 time 2014-04-28 17:42:36.753156 > msg/Pipe.cc: 1070: FAILED assert(m) > > ceph version 0.80-rc1-16-g2708c3c > (2708c3c559d99e6f3b557ee1d223efa3745f655c) > 1: (Pipe::connect()+0x3b61) [0xc18ab1] > 2: (Pipe::writer()+0x65f) [0xc193ef] > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] > 5: (clone()+0x6d) [0x7f6b089523fd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > -1> 2014-04-28 17:42:36.763322 7f6ab9792700 2 -- > 10.76.0.44:6814/192022485 >> 10.76.0.43:6808/64004052 pipe(0x43773b80 sd=121 > :31644 s=1 pgs=0 cs=0 l=0 c=0x70e3f600). got newly_acked_seq 268 vs out_seq > 0 > 0> 2014-04-28 17:42:36.765117 7f6ab9792700 -1 msg/Pipe.cc: In function > 'int Pipe::connect()' thread 7f6ab9792700 time 2014-04-28 17:42:36.763390 > msg/Pipe.cc: 1070: FAILED assert(m) > > ceph version 0.80-rc1-16-g2708c3c > (2708c3c559d99e6f3b557ee1d223efa3745f655c) > 1: (Pipe::connect()+0x3b61) [0xc18ab1] > 2: (Pipe::writer()+0x65f) [0xc193ef] > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] > 5: (clone()+0x6d) [0x7f6b089523fd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 keyvaluestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.19.log > --- end dump of recent events --- > --- begin dump of recent events --- > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 keyvaluestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.19.log > --- end dump of recent events --- > --- begin dump of recent events --- > --- logging levels --- > > > Log of another osd: > -100> 2014-04-28 19:22:13.145419 7f5a3bbdb700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.6:6805/25058567 pipe(0x5e227400 sd=55 :18619 s=4 pgs=754 cs=1 l=1 > c=0x1fea8840).reader couldn't read tag, (0) Success > -99> 2014-04-28 19:22:13.145473 7f5a3bbdb700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.6:6805/25058567 pipe(0x5e227400 sd=55 :18619 s=4 pgs=754 cs=1 l=1 > c=0x1fea8840).fault (0) Success > -98> 2014-04-28 19:22:13.145494 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.4:6829/5061602 0x1eba0f00 > -97> 2014-04-28 19:22:13.145526 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.5:6807/190001647 0x10af2000 > -96> 2014-04-28 19:22:13.145500 7f5a486f9700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6819/192003077 pipe(0x3f309180 sd=212 :3321 s=4 pgs=5467 cs=1 l=1 > c=0x1c62e9a0).reader couldn't read tag, (0) Success > -95> 2014-04-28 19:22:13.145557 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.5:6826/192003077 0x10af2280 > -94> 2014-04-28 19:22:13.145558 7f5a486f9700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6819/192003077 pipe(0x3f309180 sd=212 :3321 s=4 pgs=5467 cs=1 l=1 > c=0x1c62e9a0).fault (0) Success > -93> 2014-04-28 19:22:13.145540 7f5a38fa9700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6829/5061602 pipe(0x1eba0f00 sd=102 :14027 s=4 pgs=4582 cs=1 l=1 > c=0x43269760).reader couldn't read tag, (0) Success > -92> 2014-04-28 19:22:13.145457 7f5a62f65700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6808/66054480 pipe(0x4ee41900 sd=166 :55083 s=4 pgs=2844 cs=1 l=1 > c=0x272e3e40).reader couldn't read tag, (0) Success > -91> 2014-04-28 19:22:13.145590 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.42:6812/3051038 0x4370b400 > -90> 2014-04-28 19:22:13.145592 7f5a38fa9700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6829/5061602 pipe(0x1eba0f00 sd=102 :14027 s=4 pgs=4582 cs=1 l=1 > c=0x43269760).fault (0) Success > -89> 2014-04-28 19:22:13.145622 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.4:6825/3061991 0x1eb7c280 > -88> 2014-04-28 19:22:13.145585 7f5a3ef56700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6807/190001647 pipe(0x10af2000 sd=224 :7489 s=4 pgs=5188 cs=1 > l=1 c=0x1eae4580).reader couldn't read tag, (0) Success > -87> 2014-04-28 19:22:13.145596 7f5a62f65700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6808/66054480 pipe(0x4ee41900 sd=166 :55083 s=4 pgs=2844 cs=1 l=1 > c=0x272e3e40).fault (0) Success > -86> 2014-04-28 19:22:13.145641 7f5a3ef56700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6807/190001647 pipe(0x10af2000 sd=224 :7489 s=4 pgs=5188 cs=1 > l=1 c=0x1eae4580).fault (0) Success > -85> 2014-04-28 19:22:13.145606 7f5a4822a700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6826/192003077 pipe(0x10af2280 sd=213 :22370 s=4 pgs=5479 cs=1 > l=1 c=0x1c62ec60).reader couldn't read tag, (0) Success > -84> 2014-04-28 19:22:13.145661 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.4:6827/3051038 0x4370b180 > -83> 2014-04-28 19:22:13.145633 7f5a3bcdc700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6812/3051038 pipe(0x4370b400 sd=130 :31567 s=4 pgs=4603 cs=1 l=1 > c=0x34197600).reader couldn't read tag, (0) Success > -82> 2014-04-28 19:22:13.145660 7f5a6160a700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6825/3061991 pipe(0x1eb7c280 sd=191 :36486 s=4 pgs=4601 cs=1 l=1 > c=0x3f903080).reader couldn't read tag, (0) Success > -81> 2014-04-28 19:22:13.145695 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.4:6807/5060598 0x1eba0500 > -80> 2014-04-28 19:22:13.145705 7f5a6160a700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6825/3061991 pipe(0x1eb7c280 sd=191 :36486 s=4 pgs=4601 cs=1 l=1 > c=0x3f903080).fault (0) Success > -79> 2014-04-28 19:22:13.145699 7f5a66fa4700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6827/3051038 pipe(0x4370b180 sd=75 :48974 s=4 pgs=4568 cs=1 l=1 > c=0x3b891600).reader couldn't read tag, (0) Success > -78> 2014-04-28 19:22:13.145684 7f5a3bcdc700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6812/3051038 pipe(0x4370b400 sd=130 :31567 s=4 pgs=4603 cs=1 l=1 > c=0x34197600).fault (0) Success > -77> 2014-04-28 19:22:13.145730 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.5:6818/184055098 0x43d2d180 > -76> 2014-04-28 19:22:13.145767 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.4:6823/2060205 0x1eb7d400 > -75> 2014-04-28 19:22:13.145737 7f5a3e877700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6807/5060598 pipe(0x1eba0500 sd=92 :9724 s=4 pgs=4362 cs=1 l=1 > c=0x4556a9a0).reader couldn't read tag, (0) Success > -74> 2014-04-28 19:22:13.145734 7f5a66fa4700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6827/3051038 pipe(0x4370b180 sd=75 :48974 s=4 pgs=4568 cs=1 l=1 > c=0x3b891600).fault (0) Success > -73> 2014-04-28 19:22:13.145783 7f5a3e877700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6807/5060598 pipe(0x1eba0500 sd=92 :9724 s=4 pgs=4362 cs=1 l=1 > c=0x4556a9a0).fault (0) Success > -72> 2014-04-28 19:22:13.145664 7f5a4822a700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6826/192003077 pipe(0x10af2280 sd=213 :22370 s=4 pgs=5479 cs=1 > l=1 c=0x1c62ec60).fault (0) Success > -71> 2014-04-28 19:22:13.145799 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.4:6820/4050662 0x4370b680 > -70> 2014-04-28 19:22:13.145773 7f5a36228700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6818/184055098 pipe(0x43d2d180 sd=245 :1882 s=4 pgs=6058 cs=1 > l=1 c=0x78da4420).reader couldn't read tag, (0) Success > -69> 2014-04-28 19:22:13.145813 7f5a36228700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6818/184055098 pipe(0x43d2d180 sd=245 :1882 s=4 pgs=6058 cs=1 > l=1 c=0x78da4420).fault (0) Success > -68> 2014-04-28 19:22:13.145805 7f5a41fc9700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6823/2060205 pipe(0x1eb7d400 sd=219 :17008 s=4 pgs=4258 cs=1 l=1 > c=0x55b81a20).reader couldn't read tag, (0) Success > -67> 2014-04-28 19:22:13.145831 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.43:6821/184055098 0x10af3b80 > -66> 2014-04-28 19:22:13.145839 7f5a41fc9700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6823/2060205 pipe(0x1eb7d400 sd=219 :17008 s=4 pgs=4258 cs=1 l=1 > c=0x55b81a20).fault (0) Success > -65> 2014-04-28 19:22:13.145864 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.43:6816/75004052 0x40d48780 > -64> 2014-04-28 19:22:13.145860 7f5a2f5e7700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6820/4050662 pipe(0x4370b680 sd=57 :2505 s=4 pgs=5020 cs=1 l=1 > c=0x2d62edc0).reader couldn't read tag, (0) Success > -63> 2014-04-28 19:22:13.145892 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.5:6812/66054480 0x9ee16500 > -62> 2014-04-28 19:22:13.145872 7f5a2f6e8700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6821/184055098 pipe(0x10af3b80 sd=206 :23018 s=4 pgs=6057 cs=1 > l=1 c=0x63c0a160).reader couldn't read tag, (0) Success > -61> 2014-04-28 19:22:13.145897 7f5a60be1700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6816/75004052 pipe(0x40d48780 sd=181 :27019 s=4 pgs=2921 cs=1 l=1 > c=0x3f902420).reader couldn't read tag, (0) Success > -60> 2014-04-28 19:22:13.145950 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.43:6823/24030366 0x5469aa00 > -59> 2014-04-28 19:22:13.145936 7f5a2f6e8700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6821/184055098 pipe(0x10af3b80 sd=206 :23018 s=4 pgs=6057 cs=1 > l=1 c=0x63c0a160).fault (0) Success > -58> 2014-04-28 19:22:13.145959 7f5a60be1700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6816/75004052 pipe(0x40d48780 sd=181 :27019 s=4 pgs=2921 cs=1 l=1 > c=0x3f902420).fault (0) Success > -57> 2014-04-28 19:22:13.145951 7f5a62747700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6812/66054480 pipe(0x9ee16500 sd=164 :14615 s=4 pgs=2863 cs=1 > l=1 c=0x3f903340).reader couldn't read tag, (0) Success > -56> 2014-04-28 19:22:13.145979 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.4:6815/4056978 0x4370ac80 > -55> 2014-04-28 19:22:13.145981 7f5a62747700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6812/66054480 pipe(0x9ee16500 sd=164 :14615 s=4 pgs=2863 cs=1 > l=1 c=0x3f903340).fault (0) Success > -54> 2014-04-28 19:22:13.146029 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.5:6816/74002481 0x3f308000 > -53> 2014-04-28 19:22:13.145898 7f5a2f5e7700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6820/4050662 pipe(0x4370b680 sd=57 :2505 s=4 pgs=5020 cs=1 l=1 > c=0x2d62edc0).fault (0) Success > -52> 2014-04-28 19:22:13.146059 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.43:6814/186056525 0x4ee40c80 > -51> 2014-04-28 19:22:13.145991 7f5a65a5b700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6823/24030366 pipe(0x5469aa00 sd=31 :56476 s=4 pgs=1294 cs=1 l=1 > c=0x4b0de580).reader couldn't read tag, (0) Success > -50> 2014-04-28 19:22:13.146061 7f5a30000700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6815/4056978 pipe(0x4370ac80 sd=58 :5673 s=4 pgs=4629 cs=1 l=1 > c=0x3b8918c0).reader couldn't read tag, (0) Success > -49> 2014-04-28 19:22:13.146072 7f5a640c6700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6816/74002481 pipe(0x3f308000 sd=208 :61372 s=4 pgs=3339 cs=1 > l=1 c=0xb732dc0).reader couldn't read tag, (0) Success > -48> 2014-04-28 19:22:13.146096 7f5a30000700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6815/4056978 pipe(0x4370ac80 sd=58 :5673 s=4 pgs=4629 cs=1 l=1 > c=0x3b8918c0).fault (0) Success > -47> 2014-04-28 19:22:13.146081 7f5a65a5b700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6823/24030366 pipe(0x5469aa00 sd=31 :56476 s=4 pgs=1294 cs=1 l=1 > c=0x4b0de580).fault (0) Success > -46> 2014-04-28 19:22:13.146103 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.42:6806/4056978 0x4370af00 > -45> 2014-04-28 19:22:13.146106 7f5a640c6700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6816/74002481 pipe(0x3f308000 sd=208 :61372 s=4 pgs=3339 cs=1 > l=1 c=0xb732dc0).fault (0) Success > -44> 2014-04-28 19:22:13.146139 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.5:6815/186056525 0x4ee40780 > -43> 2014-04-28 19:22:13.146171 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.42:6817/5061602 0x1eba0a00 > -42> 2014-04-28 19:22:13.146202 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.43:6801/190001647 0x10af2f00 > -41> 2014-04-28 19:22:13.146186 7f5a67cd2700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6806/4056978 pipe(0x4370af00 sd=82 :3622 s=4 pgs=4646 cs=1 l=1 > c=0xd154c60).reader couldn't read tag, (0) Success > -40> 2014-04-28 19:22:13.146182 7f5a62646700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6815/186056525 pipe(0x4ee40780 sd=176 :54924 s=4 pgs=6348 cs=1 > l=1 c=0x42c33760).reader couldn't read tag, (0) Success > -39> 2014-04-28 19:22:13.146184 7f5a62b56700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6814/186056525 pipe(0x4ee40c80 sd=167 :19387 s=4 pgs=6339 cs=1 > l=1 c=0xd155080).reader couldn't read tag, (0) Success > -38> 2014-04-28 19:22:13.146213 7f5a67cd2700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6806/4056978 pipe(0x4370af00 sd=82 :3622 s=4 pgs=4646 cs=1 l=1 > c=0xd154c60).fault (0) Success > -37> 2014-04-28 19:22:13.146218 7f5a62646700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6815/186056525 pipe(0x4ee40780 sd=176 :54924 s=4 pgs=6348 cs=1 > l=1 c=0x42c33760).fault (0) Success > -36> 2014-04-28 19:22:13.146225 7f5a62b56700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6814/186056525 pipe(0x4ee40c80 sd=167 :19387 s=4 pgs=6339 cs=1 > l=1 c=0xd155080).fault (0) Success > -35> 2014-04-28 19:22:13.146231 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.44:6803/25058567 0x5e227900 > -34> 2014-04-28 19:22:13.146210 7f5a662d7700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6817/5061602 pipe(0x1eba0a00 sd=101 :1604 s=4 pgs=4605 cs=1 l=1 > c=0x62e22580).reader couldn't read tag, (0) Success > -33> 2014-04-28 19:22:13.146269 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.42:6802/5060598 0x4370a000 > -32> 2014-04-28 19:22:13.146241 7f5a3f360700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6801/190001647 pipe(0x10af2f00 sd=223 :48654 s=4 pgs=5173 cs=1 > l=1 c=0xd4ca000).reader couldn't read tag, (0) Success > -31> 2014-04-28 19:22:13.146269 7f5a662d7700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6817/5061602 pipe(0x1eba0a00 sd=101 :1604 s=4 pgs=4605 cs=1 l=1 > c=0x62e22580).fault (0) Success > -30> 2014-04-28 19:22:13.146297 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.5:6803/75004052 0x40d48c80 > -29> 2014-04-28 19:22:13.146285 7f5a3f360700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6801/190001647 pipe(0x10af2f00 sd=223 :48654 s=4 pgs=5173 cs=1 > l=1 c=0xd4ca000).fault (0) Success > -28> 2014-04-28 19:22:13.146278 7f5a31b96700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.44:6803/25058567 pipe(0x5e227900 sd=49 :52723 s=4 pgs=752 cs=1 l=1 > c=0xa5f22c0).reader couldn't read tag, (0) Success > -27> 2014-04-28 19:22:13.146326 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.43:6805/74002481 0x4c3e9180 > -26> 2014-04-28 19:22:13.146327 7f5a31b96700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.44:6803/25058567 pipe(0x5e227900 sd=49 :52723 s=4 pgs=752 cs=1 l=1 > c=0xa5f22c0).fault (0) Success > -25> 2014-04-28 19:22:13.146309 7f5a64bf1700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6802/5060598 pipe(0x4370a000 sd=90 :37186 s=4 pgs=4370 cs=1 l=1 > c=0x122afa20).reader couldn't read tag, (0) Success > -24> 2014-04-28 19:22:13.146353 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.42:6810/2060205 0x1eb7db80 > -23> 2014-04-28 19:22:13.146359 7f5a64bf1700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6802/5060598 pipe(0x4370a000 sd=90 :37186 s=4 pgs=4370 cs=1 l=1 > c=0x122afa20).fault (0) Success > -22> 2014-04-28 19:22:13.146381 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.42:6809/4050662 0x4370b900 > -21> 2014-04-28 19:22:13.146341 7f5a6202a700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6803/75004052 pipe(0x40d48c80 sd=187 :4544 s=4 pgs=2936 cs=1 l=1 > c=0x3f902dc0).reader couldn't read tag, (0) Success > -20> 2014-04-28 19:22:13.146398 7f5a6202a700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6803/75004052 pipe(0x40d48c80 sd=187 :4544 s=4 pgs=2936 cs=1 l=1 > c=0x3f902dc0).fault (0) Success > -19> 2014-04-28 19:22:13.146411 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.5:6820/24030366 0x34c8f00 > -18> 2014-04-28 19:22:13.146393 7f5a41bd4700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6810/2060205 pipe(0x1eb7db80 sd=183 :16386 s=4 pgs=4253 cs=1 l=1 > c=0x55b818c0).reader couldn't read tag, (0) Success > -17> 2014-04-28 19:22:13.146445 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.42:6831/3061991 0x40d49400 > -16> 2014-04-28 19:22:13.146421 7f5a661d6700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6805/74002481 pipe(0x4c3e9180 sd=207 :25307 s=4 pgs=3343 cs=1 l=1 > c=0x7cc3e40).reader couldn't read tag, (0) Success > -15> 2014-04-28 19:22:13.146419 7f5a6dc17700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6809/4050662 pipe(0x4370b900 sd=132 :3860 s=4 pgs=5071 cs=1 l=1 > c=0x3b890b00).reader couldn't read tag, (0) Success > -14> 2014-04-28 19:22:13.146437 7f5a41bd4700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6810/2060205 pipe(0x1eb7db80 sd=183 :16386 s=4 pgs=4253 cs=1 l=1 > c=0x55b818c0).fault (0) Success > -13> 2014-04-28 19:22:13.146460 7f5a6dc17700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6809/4050662 pipe(0x4370b900 sd=132 :3860 s=4 pgs=5071 cs=1 l=1 > c=0x3b890b00).fault (0) Success > -12> 2014-04-28 19:22:13.146451 7f5a3b03d700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6820/24030366 pipe(0x34c8f00 sd=28 :2684 s=4 pgs=1300 cs=1 l=1 > c=0x4b0df080).reader couldn't read tag, (0) Success > -11> 2014-04-28 19:22:13.146473 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 10.76.0.42:6815/3061091 0x1eba0280 > -10> 2014-04-28 19:22:13.146459 7f5a661d6700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.43:6805/74002481 pipe(0x4c3e9180 sd=207 :25307 s=4 pgs=3343 cs=1 l=1 > c=0x7cc3e40).fault (0) Success > -9> 2014-04-28 19:22:13.146504 7f5a5d627700 5 -- 192.168.1.6:0/21280 > mark_down_all 192.168.1.4:6828/3061091 0x1eba1680 > -8> 2014-04-28 19:22:13.146545 7f5a5b623700 1 -- 192.168.1.6:0/21280 > mark_down 0xa5f22c0 -- pipe dne > -7> 2014-04-28 19:22:13.146512 7f5a387a1700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6815/3061091 pipe(0x1eba0280 sd=93 :24347 s=4 pgs=4092 cs=1 l=1 > c=0x1e12d8c0).reader couldn't read tag, (0) Success > -6> 2014-04-28 19:22:13.146478 7f5a64af0700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6831/3061991 pipe(0x40d49400 sd=189 :57711 s=4 pgs=4609 cs=1 l=1 > c=0x3f903600).reader couldn't read tag, (0) Success > -5> 2014-04-28 19:22:13.146545 7f5a3d0c2700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6828/3061091 pipe(0x1eba1680 sd=95 :13902 s=4 pgs=4061 cs=1 l=1 > c=0x7cb4160).reader couldn't read tag, (0) Success > -4> 2014-04-28 19:22:13.146580 7f5a64af0700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6831/3061991 pipe(0x40d49400 sd=189 :57711 s=4 pgs=4609 cs=1 l=1 > c=0x3f903600).fault (0) Success > -3> 2014-04-28 19:22:13.146560 7f5a387a1700 2 -- 192.168.1.6:0/21280 >> > 10.76.0.42:6815/3061091 pipe(0x1eba0280 sd=93 :24347 s=4 pgs=4092 cs=1 l=1 > c=0x1e12d8c0).fault (0) Success > -2> 2014-04-28 19:22:13.146593 7f5a3d0c2700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.4:6828/3061091 pipe(0x1eba1680 sd=95 :13902 s=4 pgs=4061 cs=1 l=1 > c=0x7cb4160).fault (0) Success > -1> 2014-04-28 19:22:13.146484 7f5a3b03d700 2 -- 192.168.1.6:0/21280 >> > 192.168.1.5:6820/24030366 pipe(0x34c8f00 sd=28 :2684 s=4 pgs=1300 cs=1 l=1 > c=0x4b0df080).fault (0) Success > 0> 2014-04-28 19:22:13.245270 7f5a65cd1700 -1 *** Caught signal > (Aborted) ** > in thread 7f5a65cd1700 > > ceph version 0.80-rc1-16-g2708c3c > (2708c3c559d99e6f3b557ee1d223efa3745f655c) > 1: /usr/bin/ceph-osd() [0xa844b0] > 2: (()+0xfcb0) [0x7f5a7f493cb0] > 3: (gsignal()+0x35) [0x7f5a7d978425] > 4: (abort()+0x17b) [0x7f5a7d97bb8b] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f5a7e274e2d] > 6: (()+0x5ef26) [0x7f5a7e272f26] > 7: (()+0x5ef53) [0x7f5a7e272f53] > 8: (()+0x5f17e) [0x7f5a7e27317e] > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x43d) [0xb5e15d] > 10: (Pipe::connect()+0x3b61) [0xc18ab1] > 11: (Pipe::writer()+0x65f) [0xc193ef] > 12: (Pipe::Writer::entry()+0xd) [0xc240ad] > 13: (()+0x7e9a) [0x7f5a7f48be9a] > 14: (clone()+0x6d) [0x7f5a7da363fd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 keyvaluestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.18.log > --- end dump of recent events --- > > please help me to fix this issue and let me known if you need more > information. > > Best regards, > Thanh Tran > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
