On Thu, Jan 24, 2013 at 8:39 AM, Sage Weil <s...@inktank.com> wrote: > On Thu, 24 Jan 2013, Andrey Korolyov wrote: >> On Thu, Jan 24, 2013 at 12:59 AM, Jens Kristian S?gaard >> <j...@mermaidconsulting.dk> wrote: >> > Hi Sage, >> > >> >>> I think the problem now is just that 'osd target transaction size' is >> >> >> >> I set it to 50, and that seems to have solved all my problems. >> >> >> >> After a day or so my cluster got to a HEALTH_OK state again. It has been >> >> running for a few days now without any crashes! >> > >> > >> > Hmm, one of the OSDs crashed again, sadly. >> > >> > It logs: >> > >> > -2> 2013-01-23 18:01:23.563624 7f67524da700 1 heartbeat_map is_healthy >> > 'FileStore::op_tp thread 0x7f673affd700' had timed out after 60 >> > -1> 2013-01-23 18:01:23.563657 7f67524da700 1 heartbeat_map is_healthy >> > 'FileStore::op_tp thread 0x7f673affd700' had suicide timed out after 180 >> > 0> 2013-01-23 18:01:24.257996 7f67524da700 -1 common/HeartbeatMap.cc: >> > In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, >> > const char*, time_t)' thread 7f67524da700 time 2013-01-23 18:01:23.563677 >> > >> > common/HeartbeatMap.cc: 78: FAILED assert(0 == "hit suicide timeout") >> > >> > >> > With this stack trace: >> > >> > ceph version 0.56.1-26-g3bd8f6b (3bd8f6b7235eb14cab778e3c6dcdc636aff4f539) >> > 1: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, >> > long)+0x2eb) [0x846ecb] >> > 2: (ceph::HeartbeatMap::is_healthy()+0x8e) [0x8476ae] >> > 3: (ceph::HeartbeatMap::check_touch_file()+0x28) [0x8478d8] >> > 4: (CephContextServiceThread::entry()+0x55) [0x8e0f45] >> > 5: /lib64/libpthread.so.0() [0x3cbc807d14] >> > 6: (clone()+0x6d) [0x3cbc0f167d] >> > >> > >> > I have saved the core file, if there's anything in there you need? >> > >> > Or do you think I just need to set the target transaction size even lower >> > than 50? >> > >> > >> >> I was able to catch this too on rejoin to very busy cluster and seems >> I need to lower this value at least at start time. Also >> c5fe0965572c074a2a33660719ce3222d18c1464 has increased overall time >> before restarted or new osd will join a cluster, and for 2M objects/3T >> of replicated data restart of the cluster was took almost a hour >> before it actually begins to work. The worst thing is that a single >> osd, if restarted, will mark as up after couple of minutes, then after >> almost half of hour(eating 100 percent of one cpu, ) as down and then >> cluster will start to redistribute data after 300s timeout, osd still >> doing something. > > Okay, something is very wrong. Can you reproduce this with a log? Or > even a partial log while it is spinning? You can adjust the log level on > a running process with > > ceph --admin-daemon /var/run/ceph-osd.NN.asok config set debug_osd 20 > ceph --admin-daemon /var/run/ceph-osd.NN.asok config set debug_ms 1 > > We haven't been able to reproduce this, so I'm very much interested in any > light you can shine here. >
Unfortunately cluster finally hit ``suicide timeout'' by every osd, so there was no logs, only some backtraces[1]. Yesterday after an osd was not able to join cluster in a hour, I decided to wait until data is remapped, then tried to restart cluster, leaving it overnight, to morning all osd processes are dead, with the same backtraces. Before it, after a silly node crash(related to deadlocks in kernel kvm code), some pgs remains to stay in peering state without any blocker in json output, so I had decided to restart osd to which primary copy belongs, because it helped before. So most interesting part is missing, but I`ll reformat cluster soon and will try to catch this again after filling some data in. [1]. http://xdel.ru/downloads/ceph-log/osd-heartbeat/ > Thanks! > sage > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
