----- Message from Gregory Farnum <[email protected]> ---------
Date: Tue, 1 Apr 2014 09:03:17 -0700
From: Gregory Farnum <[email protected]>
Subject: Re: [ceph-users] ceph 0.78 mon and mds crashing (bus error)
To: "Yan, Zheng" <[email protected]>
Cc: Kenneth Waegeman <[email protected]>, ceph-users
<[email protected]>
On Tue, Apr 1, 2014 at 7:12 AM, Yan, Zheng <[email protected]> wrote:
On Tue, Apr 1, 2014 at 10:02 PM, Kenneth Waegeman
<[email protected]> wrote:
After some more searching, I've found that the source of the
problem is with
the mds and not the mon.. The mds crashes, generates a core dump that eats
the local space, and in turn the monitor (because of leveldb) crashes.
The error in the mds log of one host:
2014-04-01 15:46:34.414615 7f870e319700 0 -- 10.141.8.180:6836/13152 >>
10.141.8.180:6789/0 pipe(0x517371180 sd=54 :42439 s=4 pgs=0 cs=0 l=1
c=0x147ac780).connect got RESETSESSION but no longer connecting
2014-04-01 15:46:34.438792 7f871194f700 0 -- 10.141.8.180:6836/13152 >>
10.141.8.180:6789/0 pipe(0x1b099f580 sd=8 :43150 s=4 pgs=0 cs=0 l=1
c=0x1fd44360).connect got RESETSESSION but no longer connecting
2014-04-01 15:46:34.439028 7f870e319700 0 -- 10.141.8.180:6836/13152 >>
10.141.8.182:6789/0 pipe(0x13aa64880 sd=54 :37085 s=4 pgs=0 cs=0 l=1
c=0x1fd43de0).connect got RESETSESSION but no longer connecting
2014-04-01 15:46:34.468257 7f871b7ae700 -1 mds/CDir.cc: In function 'void
CDir::_omap_fetched(ceph::bufferlist&, std::map<std::basic_string<char,
std::char_traits<char>, std::allocator<char> >, ceph::buffer::list,
std::less<std::basic_string<char, std::char_traits<char>,
std::allocator<char> > >, std::allocator<std::pair<const
std::basic_string<char, std::char_traits<char>, std::allocator<char> >,
ceph::buffer::list> > >&, const std::string&, int)' thread
7f871b7ae700 time
2014-04-01 15:46:34.448320
mds/CDir.cc: 1474: FAILED assert(r == 0 || r == -2 || r == -61)
could you use gdb to check what is value of variable 'r' .
If you look at the crash dump log you can see the return value in the
osd_op_reply message:
-1> 2014-04-01 15:46:34.440860 7f871b7ae700 1 --
10.141.8.180:6836/13152 <== osd.3 10.141.8.180:6827/4366 33077 ====
osd_op_reply(4179177 100001f2ef1.00000000 [omap-get-header
0~0,omap-get-vals 0~16] v0'0 uv0 ack = -108 (Cannot send after
transport endpoint shutdown)) v6 ==== 229+0+0 (958358678 0 0)
0x2cff7aa80 con 0x37ea3c0
-108, which is ESHUTDOWN, but we also use it (via the 108 constant, I
think because ESHUTDOWN varies across platforms) as EBLACKLISTED.
So it looks like this is itself actually a symptom of another problem
that is causing the MDS to get timed out on the monitor. If a core
dump is "eating the local space", maybe the MDS is stuck in an
infinite allocation loop of some kind? How big are your disks,
Kenneth? Do you have any information on how much CPU/memory the MDS
was using before this?
I monitored the mds process after restart:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19215 root 20 0 6070m 5.7g 5236 S 778.6 18.1 1:27.54 ceph-mds
19215 root 20 0 7926m 7.5g 5236 S 179.2 23.8 2:44.39 ceph-mds
19215 root 20 0 12.4g 12g 5236 S 157.2 38.8 3:43.47 ceph-mds
19215 root 20 0 16.6g 16g 5236 S 144.4 52.0 4:15.01 ceph-mds
19215 root 20 0 19.9g 19g 5236 S 137.2 62.5 4:35.83 ceph-mds
19215 root 20 0 24.5g 24g 5224 S 136.5 77.0 5:04.66 ceph-mds
19215 root 20 0 25.8g 25g 2944 S 33.7 81.2 5:13.74 ceph-mds
19215 root 20 0 26.0g 25g 2916 S 24.6 81.7 5:19.07 ceph-mds
19215 root 20 0 26.1g 25g 2916 S 13.0 82.1 5:22.16 ceph-mds
19215 root 20 0 27.7g 26g 1856 S 100.0 85.8 5:36.46 ceph-mds
Then it crashes. I changed the core dump location out of the root fs,
the core dump is indeed about 26G
My disks:
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 9.9G 2.9G 6.5G 31% /
tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 248M 53M 183M 23% /boot
/dev/sda4 172G 61G 112G 35% /var/lib/ceph/log/sda4
/dev/sdb 187G 61G 127G 33% /var/lib/ceph/log/sdb
/dev/sdc 3.7T 1.7T 2.0T 47% /var/lib/ceph/osd/sdc
/dev/sdd 3.7T 1.5T 2.2T 41% /var/lib/ceph/osd/sdd
/dev/sde 3.7T 1.4T 2.4T 37% /var/lib/ceph/osd/sde
/dev/sdf 3.7T 1.5T 2.3T 39% /var/lib/ceph/osd/sdf
/dev/sdg 3.7T 2.1T 1.7T 56% /var/lib/ceph/osd/sdg
/dev/sdh 3.7T 1.7T 2.0T 47% /var/lib/ceph/osd/sdh
/dev/sdi 3.7T 1.7T 2.0T 47% /var/lib/ceph/osd/sdi
/dev/sdj 3.7T 1.7T 2.0T 47% /var/lib/ceph/osd/sdj
/dev/sdk 3.7T 2.1T 1.6T 58% /var/lib/ceph/osd/sdk
/dev/sdl 3.7T 1.7T 2.0T 46% /var/lib/ceph/osd/sdl
/dev/sdm 3.7T 1.5T 2.2T 41% /var/lib/ceph/osd/sdm
/dev/sdn 3.7T 1.4T 2.3T 38% /var/lib/ceph/osd/sdn
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
----- End message from Gregory Farnum <[email protected]> -----
--
Met vriendelijke groeten,
Kenneth Waegeman
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
