Hello, One of the osd in my ceph cluster, change to down and autoout, I did not get the root cause in the osd log. Could you help?
2014-05-30 17:35:55.541353 7f7b03a937a0 0 ceph version 0.80 (b78644e7dee100e48dfeca32c9270a6b210d3003), process ceph-osd, pid 5519 2014-05-30 17:35:55.544601 7f7b03a937a0 0 filestore(/var/lib/ceph/osd/osd11) mount detected xfs (libxfs) 2014-05-30 17:35:55.544612 7f7b03a937a0 1 filestore(/var/lib/ceph/osd/osd11) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2014-05-30 17:35:55.611316 7f7b03a937a0 0 genericfilestorebackend(/var/lib/ceph/osd/osd11) detect_features: FIEMAP ioctl is supported and appears to work 2014-05-30 17:35:55.611335 7f7b03a937a0 0 genericfilestorebackend(/var/lib/ceph/osd/osd11) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2014-05-30 17:35:55.620215 7f7b03a937a0 0 genericfilestorebackend(/var/lib/ceph/osd/osd11) detect_features: syscall(SYS_syncfs, fd) fully supported 2014-05-30 17:35:55.620290 7f7b03a937a0 0 xfsfilestorebackend(/var/lib/ceph/osd/osd11) detect_feature: extsize is supported 2014-05-30 17:35:55.720679 7f7b03a937a0 0 filestore(/var/lib/ceph/osd/osd11) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2014-05-30 17:35:55.725759 7f7b03a937a0 1 journal _open /dev/disk/by-path/pci-0000:08:00.0-sas-0x4433221100000000-lun-0-part6 fd 20: 1085702144 bytes, block size 4096 bytes, directio = 1, ai o = 1 2014-05-30 17:35:55.729304 7f7b03a937a0 1 journal _open /dev/disk/by-path/pci-0000:08:00.0-sas-0x4433221100000000-lun-0-part6 fd 20: 1085702144 bytes, block size 4096 bytes, directio = 1, ai o = 1 2014-05-30 17:35:55.729724 7f7b03a937a0 1 journal close /dev/disk/by-path/pci-0000:08:00.0-sas-0x4433221100000000-lun-0-part6 2014-05-30 17:35:55.730358 7f7b03a937a0 0 filestore(/var/lib/ceph/osd/osd11) mount detected xfs (libxfs) 2014-05-30 17:35:55.770762 7f7b03a937a0 0 genericfilestorebackend(/var/lib/ceph/osd/osd11) detect_features: FIEMAP ioctl is supported and appears to work 2014-05-30 17:35:55.770778 7f7b03a937a0 0 genericfilestorebackend(/var/lib/ceph/osd/osd11) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2014-05-30 17:35:55.803697 7f7b03a937a0 0 genericfilestorebackend(/var/lib/ceph/osd/osd11) detect_features: syscall(SYS_syncfs, fd) fully supported 2014-05-30 17:35:55.803767 7f7b03a937a0 0 xfsfilestorebackend(/var/lib/ceph/osd/osd11) detect_feature: extsize is supported 2014-05-30 17:35:55.870678 7f7b03a937a0 0 filestore(/var/lib/ceph/osd/osd11) mount: WRITEAHEAD journal mode explicitly enabled in conf 2014-05-30 17:35:55.873916 7f7b03a937a0 1 journal _open /dev/disk/by-path/pci-0000:08:00.0-sas-0x4433221100000000-lun-0-part6 fd 21: 1085702144 bytes, block size 4096 bytes, directio = 1, ai o = 1 2014-05-30 17:35:55.877497 7f7b03a937a0 1 journal _open /dev/disk/by-path/pci-0000:08:00.0-sas-0x4433221100000000-lun-0-part6 fd 21: 1085702144 bytes, block size 4096 bytes, directio = 1, ai o = 1 2014-05-30 17:35:55.887357 7f7b03a937a0 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello 2014-05-30 17:35:55.890557 7f7b03a937a0 0 osd.11 0 crush map has features 33816576, adjusting msgr requires for clients 2014-05-30 17:35:55.890566 7f7b03a937a0 0 osd.11 0 crush map has features 33816576, adjusting msgr requires for osds 2014-05-30 17:35:55.890584 7f7b03a937a0 0 osd.11 0 load_pgs 2014-05-30 17:35:55.890616 7f7b03a937a0 0 osd.11 0 load_pgs opened 0 pgs 2014-05-30 17:35:55.891715 7f7b03a77700 0 -- 192.168.1.4:6803/5519 >> 192.168.0.3:6789/0 pipe(0x3908000 sd=25 :0 s=1 pgs=0 cs=0 l=1 c=0x381dc20).fault 2014-05-30 17:36:10.894794 7f7af4107700 0 osd.11 0 ignoring osdmap until we have initialized 2014-05-30 17:36:10.894917 7f7af4107700 0 osd.11 0 ignoring osdmap until we have initialized 2014-05-30 17:36:10.895006 7f7b03a937a0 0 osd.11 0 done with init, starting boot process 2014-05-30 17:36:11.932943 7f7af4107700 0 osd.11 4 crush map has features 1107558400, adjusting msgr requires for clients 2014-05-30 17:36:11.932946 7f7af4107700 0 osd.11 4 crush map has features 1107558400, adjusting msgr requires for osds 2014-05-30 17:36:16.219495 7f7af3105700 0 osd.11 7 crush map has features 2200130813952, adjusting msgr requires for clients 2014-05-30 17:36:16.219503 7f7af3105700 0 osd.11 7 crush map has features 2200130813952, adjusting msgr requires for osds 2014-05-30 17:38:18.193556 7f7af4107700 0 monclient: hunting for new mon 2014-05-30 17:39:07.988294 7f7ae1086700 0 -- 192.168.21.4:6802/5519 >> 192.168.20.5:6800/29760 pipe(0x390f300 sd=102 :6802 s=0 pgs=0 cs=0 l=0 c=0x42489a0).accept connect_seq 0 vs existing 0 state wait 2014-05-30 17:45:06.232425 7f7ae1086700 0 -- 192.168.21.4:6802/5519 >> 192.168.20.5:6800/29760 pipe(0x390f300 sd=102 :6802 s=2 pgs=10 cs=1 l=0 c=0x424dac0).fault with nothing to send, going to standby 2014-05-30 17:50:59.728170 7f7ae1389700 0 -- 192.168.21.4:6802/5519 >> 192.168.20.5:6800/3357 pipe(0x40dbc00 sd=144 :6802 s=0 pgs=0 cs=0 l=0 c=0x424ee00).accept connect_seq 0 vs existing 0 state connecting 2014-05-30 17:50:59.728250 7f7ae7fee700 0 -- 192.168.21.4:6802/5519 >> 192.168.20.5:6800/3357 pipe(0x40db480 sd=143 :46676 s=4 pgs=0 cs=0 l=0 c=0x424e880).connect got RESETSESSION but no longer connecting 2014-05-30 17:51:21.096279 7f7ae3ab0700 0 -- 192.168.21.4:6802/5519 >> 192.168.21.2:6806/5835 pipe(0x390d280 sd=70 :36622 s=2 pgs=11 cs=1 l=0 c=0x39fdc20).fault with nothing to send, going to standby 2014-05-30 18:00:05.573815 7f7ae75eb700 0 -- 192.168.21.4:6802/5519 >> 192.168.20.3:6802/30559 pipe(0x3908a00 sd=36 :6802 s=2 pgs=12 cs=1 l=0 c=0x381eb40).fault with nothing to send, going to standby 2014-05-30 18:00:05.654955 7f7ae3fb5700 0 -- 192.168.21.4:6802/5519 >> 192.168.21.2:6808/5902 pipe(0x390d780 sd=61 :50075 s=2 pgs=10 cs=1 l=0 c=0x39fe1a0).fault with nothing to send, going to standby 2014-05-30 18:00:05.656277 7f7ae41b7700 0 -- 192.168.21.4:6802/5519 >> 192.168.20.3:6800/30492 pipe(0x390da00 sd=60 :51557 s=2 pgs=11 cs=1 l=0 c=0x39fb5a0).fault with nothing to send, going to standby 2014-05-30 18:00:05.670832 7f7ae37ad700 0 -- 192.168.21.4:6802/5519 >> 192.168.20.3:6804/30626 pipe(0x390cd80 sd=73 :35960 s=2 pgs=10 cs=1 l=0 c=0x39fd6a0).fault with nothing to send, going to standby 2014-05-30 18:00:07.671926 7f7ae42b8700 0 -- 192.168.21.4:6802/5519 >> 192.168.21.2:6804/5768 pipe(0x390bc00 sd=59 :55352 s=2 pgs=13 cs=1 l=0 c=0x39fb860).fault with nothing to send, going to standby 2014-05-30 18:00:09.169280 7f7ae2fa5700 0 -- 192.168.21.4:6802/5519 >> 192.168.20.3:6806/30693 pipe(0x390f580 sd=92 :6802 s=2 pgs=6 cs=1 l=0 c=0x381eca0).fault with nothing to send, going to standby 2014-05-30 18:00:09.171271 7f7ae3db3700 0 -- 192.168.21.4:6802/5519 >> 192.168.20.3:6808/30760 pipe(0x390d500 sd=64 :51855 s=2 pgs=11 cs=1 l=0 c=0x39fdee0).fault with nothing to send, going to standby 2014-05-30 18:00:09.172818 7f7ae38ae700 0 -- 192.168.21.4:6802/5519 >> 192.168.21.2:6800/5633 pipe(0x390d000 sd=72 :56968 s=2 pgs=8 cs=1 l=0 c=0x39fd960).fault with nothing to send, going to standby 2014-05-30 18:00:10.472991 7f7ae36ac700 0 -- 192.168.21.4:6802/5519 >> 192.168.21.2:6802/5700 pipe(0x390cb00 sd=74 :46340 s=2 pgs=10 cs=1 l=0 c=0x39ff7a0).fault with nothing to send, going to standby 2014-05-30 18:01:09.118219 7f7ae0f85700 0 -- 192.168.21.4:6802/5519 >> 192.168.20.3:6804/30626 pipe(0x40db980 sd=150 :6802 s=0 pgs=0 cs=0 l=0 c=0x424eca0).accept connect_seq 2 vs existing 1 state standby 2014-05-30 18:01:10.288703 7f7ae3bb1700 0 -- 192.168.21.4:6802/5519 >> 192.168.20.3:6802/30559 pipe(0x40db700 sd=152 :6802 s=0 pgs=0 cs=0 l=0 c=0x424e9e0).accept connect_seq 2 vs existing 1 state standby 2014-05-30 18:13:42.832508 7f7ae34aa700 0 -- 192.168.21.4:6802/5519 >> 192.168.21.4:6800/5452 pipe(0x390ee00 sd=75 :35061 s=2 pgs=10 cs=1 l=0 c=0x39ff4e0).fault with nothing to send, going to standby 2014-05-30 18:13:42.832914 7f7ae51c7700 0 -- 192.168.1.4:0/5519 >> 192.168.1.4:6801/5452 pipe(0x390a300 sd=41 :0 s=1 pgs=0 cs=0 l=1 c=0x47482c0).fault 2014-05-30 18:13:42.833019 7f7ae5bd1700 0 -- 192.168.1.4:0/5519 >> 192.168.21.4:6801/5452 pipe(0x3909900 sd=42 :0 s=1 pgs=0 cs=0 l=1 c=0x4748000).fault 2014-05-30 18:13:44.060769 7f7ae51c7700 0 -- 192.168.21.4:6802/5519 >> 192.168.20.3:6800/30492 pipe(0x390fa80 sd=75 :6802 s=0 pgs=0 cs=0 l=0 c=0x4748420).accept connect_seq 2 vs existing 1 state standby 2014-05-30 18:13:44.060947 7f7ae5bd1700 0 -- 192.168.21.4:6802/5519 >> 192.168.21.2:6800/5633 pipe(0x390ee00 sd=112 :6802 s=0 pgs=0 cs=0 l=0 c=0x4748000).accept connect_seq 2 vs existing 1 state standby 2014-05-30 18:13:44.061080 7f7ae34aa700 0 -- 192.168.21.4:6802/5519 >> 192.168.21.2:6804/5768 pipe(0x3909900 sd=113 :6802 s=0 pgs=0 cs=0 l=0 c=0x47482c0).accept connect_seq 2 vs existing 1 state standby 2014-05-30 18:13:44.149273 7f7ae7eed700 -1 osd.11 37 *** Got signal Terminated *** 2014-05-30 18:13:44.149298 7f7ae7eed700 0 osd.11 37 prepare_to_stop telling mon we are shutting down 2014-05-30 18:49:08.333779 7feb5f6207a0 0 ceph version 0.80 (b78644e7dee100e48dfeca32c9270a6b210d3003), process ceph-osd, pid 4341 2014-05-30 18:49:08.333913 7feb5f6207a0 -1 ^[[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/osd11: (2) No such file or directory^[[0m 2014-05-30 18:59:07.634430 7f4a259867a0 0 ceph version 0.80 (b78644e7dee100e48dfeca32c9270a6b210d3003), process ceph-osd, pid 5161 2014-05-30 18:59:07.634527 7f4a259867a0 -1 ^[[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/osd11: (2) No such file or directory^[[0m 2014-05-30 19:02:38.458643 7fa53a9527a0 0 ceph version 0.80 (b78644e7dee100e48dfeca32c9270a6b210d3003), process ceph-osd, pid 5479 2014-05-30 19:02:38.458744 7fa53a9527a0 -1 ^[[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/osd11: (2) No such file or directory^[[0m Wei Cao (Buddy)
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
