Sorry for the delay in getting back to you. I never catch a core during the segfault of the umount.
I tried to delete the heartbeat again and the command completed successfully, but the messages are still appearing in syslog. A subsequent issue of the command brings: [EMAIL PROTECTED] ~]# ocfs2_hb_ctl -K -d /dev/dm-30 o2cb ocfs2_hb_ctl: Unable to access cluster service while stopping heartbeat Please see attached for the log you requested. Daniel > -----Original Message----- > From: Sunil Mushran [mailto:[EMAIL PROTECTED] > Sent: Thursday, October 30, 2008 5:42 PM > To: Daniel Keisling > Cc: [email protected] > Subject: Re: [Ocfs2-users] Another node is heartbeating in > our slot! errors with LUN removal/addition > > So manually stopping the heartbeat worked. > > Did you catch a coredump for the segfault during umount? > > There is a small difference in the stop heartbeat that is called > as part of umount and the one called by hand. But have not > been able to figure out the source of the segfault. > > The above coredump will help. > > The other thing you could do is run the following command when you > see the "another node heartbeating..." message. > $ for i in `seq 30` ; do date >>/tmp/hb.out ; debugfs.ocfs2 -R "hb" > /dev/dmX >>/tmp/hb.out ; sleep 1; done; > > Replace the device name with the one that is in the logs. > Email me the > output. > > Sunil > > Daniel Keisling wrote: > > Sunil, > > > > I manually killed the heartbeat via 'ocfs2_hb_ctl -K -d ... > o2cb' but it > > did not generate a core. It did, however, stop the "ERROR: Device > > "dm-30": another node is heartbeating in our slot!" messages. > > > > After remounting the device, the "ERROR: Device "dm-30": > another node is > > heartbeating in our slot!" messages return to syslog. > Unmounting the > > device brings a segfault: > > > > Oct 30 10:59:13 ausracdbd01 multipathd: dm-30: umount map (uevent) > > Oct 30 10:59:14 ausracdbd01 kernel: ocfs2_hb_ctl[11351]: segfault at > > 0000000000000000 rip 0000000000428fa0 rsp 00007fffe6710138 error 4 > > Oct 30 10:59:14 ausracdbd01 kernel: ocfs2: Unmounting > device (253,30) on > > (node 0) > > > > The "ERROR: Device "dm-30": another node is heartbeating in > our slot!" > > messages keep flowing through syslog until I manually remove the > > heartbeat. > > > > I can reproduce this over and over. > > > > TIA, > > > > Daniel > > > > > >> -----Original Message----- > >> From: Sunil Mushran [mailto:[EMAIL PROTECTED] > >> Sent: Friday, October 24, 2008 2:46 PM > >> To: Daniel Keisling > >> Cc: [email protected] > >> Subject: Re: [Ocfs2-users] Another node is heartbeating in > >> our slot! errors with LUN removal/addition > >> > >> So that's the problem. The heartbeat is not stopping because of the > >> segfault. I reviewed the code change in this tool (1.2.7 to 1.4.1) > >> and it is quite limited. As in, I have no idea as to why it is > >> segfaulting. > >> > >> Now you could stop the heartbeat manually. You have to be careful > >> though because stopping it for a mounted volume will be very > >> problematic to say the least. But one good reason to do it manually > >> would be to catch the coredump.... which could help tell > us what the > >> problem is. > >> > >> If you are up for it, do: > >> $ ulimit -c unlimited > >> $ ocfs2_hb_ctl -K -d /dev/sdX o2cb > >> > >> Run it on a umounted volume that has a reference left over. -I is > >> to view the number of references. > >> > >> Sunil > >> > >> Daniel Keisling wrote: > >> > >>> Oct 23 08:53:21 ausracdb03 kernel: > >>> > >> (2410,3):o2hb_do_disk_heartbeat:770 > >> > >>> ERROR: Device "dm-28": another node is heartbeating in our slot! > >>> > >>> [EMAIL PROTECTED] ~]# ocfs2_hb_ctl -I -d /dev/dm-28 > >>> 289FD533334645C5A88FD715FC0EEF85: 1 refs > >>> > >>> Yes, it segfaults every night (I take two snapshots per night): > >>> > >>> [EMAIL PROTECTED] log]# grep segfault /var/log/messages > >>> Oct 21 03:15:47 ausracdb03 kernel: ocfs2_hb_ctl[4197]: segfault at > >>> 0000000000000000 rip 0000000000428fa0 rsp 00007fffefd623e8 error 4 > >>> Oct 21 03:17:43 ausracdb03 kernel: ocfs2_hb_ctl[8002]: segfault at > >>> 0000000000000000 rip 0000000000428fa0 rsp 00007fff1a8f9318 error 4 > >>> Oct 21 16:43:30 ausracdb03 kernel: ocfs2_hb_ctl[16933]: > segfault at > >>> 0000000000000000 rip 0000000000428fa0 rsp 00007fff816aa558 error 4 > >>> Oct 21 16:43:31 ausracdb03 kernel: ocfs2_hb_ctl[16950]: > segfault at > >>> 0000000000000000 rip 0000000000428fa0 rsp 00007fffcb162b88 error 4 > >>> Oct 22 03:15:44 ausracdb03 kernel: ocfs2_hb_ctl[7721]: segfault at > >>> 0000000000000000 rip 0000000000428fa0 rsp 00007fff88a7efb8 error 4 > >>> Oct 22 03:17:46 ausracdb03 kernel: ocfs2_hb_ctl[11294]: > segfault at > >>> 0000000000000000 rip 0000000000428fa0 rsp 00007fff85549f68 error 4 > >>> Oct 23 03:15:51 ausracdb03 kernel: ocfs2_hb_ctl[32555]: > segfault at > >>> 0000000000000000 rip 0000000000428fa0 rsp 00007fff8fefe498 error 4 > >>> Oct 23 03:17:40 ausracdb03 kernel: ocfs2_hb_ctl[3756]: segfault at > >>> 0000000000000000 rip 0000000000428fa0 rsp 00007fff99bb25d8 error 4 > >>> Oct 24 03:15:47 ausracdb03 kernel: ocfs2_hb_ctl[15664]: > segfault at > >>> 0000000000000000 rip 0000000000428fa0 rsp 00007ffff4254aa8 error 4 > >>> Oct 24 03:17:43 ausracdb03 kernel: ocfs2_hb_ctl[18029]: > segfault at > >>> 0000000000000000 rip 0000000000428fa0 rsp 00007fff75055a78 error 4 > >>> > >>> > >>> This began when I upgraded to v1.4.1-1 from v1.2.8. > >>> > >>> Thanks, > >>> > >>> Daniel > >>> > >>> > >> > >> > > > > > ______________________________________________________________________ > > This email transmission and any documents, files or previous email > > messages attached to it may contain information that is > confidential or > > legally privileged. If you are not the intended recipient > or a person > > responsible for delivering this transmission to the > intended recipient, > > you are hereby notified that you must not read this transmission and > > that any disclosure, copying, printing, distribution or use of this > > transmission is strictly prohibited. If you have received > this transmission > > in error, please immediately notify the sender by telephone > or return email > > and delete the original transmission and its attachments > without reading > > or saving in any manner. > > > > > > > ______________________________________________________________________ This email transmission and any documents, files or previous email messages attached to it may contain information that is confidential or legally privileged. If you are not the intended recipient or a person responsible for delivering this transmission to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of this transmission is strictly prohibited. If you have received this transmission in error, please immediately notify the sender by telephone or return email and delete the original transmission and its attachments without reading or saving in any manner.
hb.out
Description: hb.out
_______________________________________________ Ocfs2-users mailing list [email protected] http://oss.oracle.com/mailman/listinfo/ocfs2-users
