Btw. IT could be my mistake, but I had a problems getting stable active/active (aka etherchannel) bonding on DELL switches. Give up after a few miracles I had with this thing.
(But it works well on Cisco swithces). ----- Original Message ----- From: "Andy Phillips" <[EMAIL PROTECTED]> To: "Sunil Mushran" <[EMAIL PROTECTED]> Cc: "ocfs2-users" <[email protected]> Sent: Monday, August 07, 2006 11:38 AM Subject: Re: [Ocfs2-users] o2net: connect to node has been idle for 10 secs > Hi, > > Thanks for the responses. Before anyone thinks its a network > problem, we're running with active/active bonding on a pair of > dedicated gigabit switches, with flow control on. None of the linux > systems, nor any of the network switches register any changes or faults > on the network. OCFS2 runs on one pair of interfaces, on one class > C network, and RAC runs on another pair of interfaces, on a different > class C network. RAC does not have a problem. > > We can also ping across the ocfs2 heartbeat lan with out > interruption and dropped packets as the heartbeat fails. > > As this system is production, we've backed off using ocfs2 > and have gone with NFS in the interim. I'm still committed to using > ocfs2, but we need a stable production platform. I'll be seeing if > I can replicate the issue on a testbed in the next few days. > > I notice from the timer values printed on the console as the > node dies that o2net_advance_rx is called without triggering the > code path that includes either o2net_process_message or > o2net_check_handshake, as the tv_timer timestamp is less than adv_start > and adv_end timestamps. So either there is an incomplete packet, > or some sort of erroneous message that comes in just before we > stop for 10 seconds, and then drop dead. > > Andy > > > > On Mon, 2006-08-07 at 11:14 -0700, Alexei_Roudnev wrote: > > Btw. > > > > Netwo0rk convergence time is BY DEFAULT 30 seconds MINIMUM. It means, than > > in no case I can garantee idle time on interconenct less than 40 seconds. > > > > Read STP Ethernet protocol, if you have questions. > > > > Our FibreChannel network have, to compare, 60 second timeout setting for > > MPIO switchover. NetApp have 40 seconds Failover time. > > Protoicols such as EGRP have 20 - 30 seconds reconvergence time (OSPF is > > faster - in some cases). > > > > So. > > > > If OCFSv2 see idle time > 10 seconds, it have not ANY RIGHT to reboot. IT > > can do anithing special, such as go to _recovery mode_, trying to reconnect > > and enasuring that other nodes can lost connection with it, but if it > > self-fence or reboots before trying for 60 - 90 seconds, it violate network > > time frames and do not make any sense. > > > > Moreover, as I said before - if we have not outstanding IO on OCFS, there is > > not much sense to reboot even if system must self-fence - it makes much more > > sense to _reconnect_ to the cluster or make some other attempt to start from > > the scratch. Else, any OCFS, even passive (not used), increase system > > instability in thousands times. > > > > ----- Original Message ----- > > From: "Andy Phillips" <[EMAIL PROTECTED]> > > To: "Sunil Mushran" <[EMAIL PROTECTED]> > > Cc: "ocfs2-users" <[email protected]> > > Sent: Thursday, August 03, 2006 10:20 AM > > Subject: Re: [Ocfs2-users] o2net: connect to node has been idle for 10 secs > > > > > > > Hello, > > > > > > Its doubly odd then. We'll need to schedule an upgrade to 1.2.3. > > > In the mean time, we've scheduled a cron job that touches a file > > > on each ocfs2 file system every 3 seconds. This should ensure a > > > constant flow of traffic assuming metadata updates travel across > > > the interconnect. > > > > > > I've noticed that there is one other person who seems to have > > > seen this problem - > > > http://oss.oracle.com/pipermail/ocfs2-users/2006-July/000612.html > > > but they were on an old version of kernel and fs code. Any idea > > > as to what the underlying cause may be if its not a dropped packet? > > > > > > Would you also mind letting me know what those two line changes were, > > > just for my own interest's sake. > > > > > > Thanks for the quick response. > > > > > > Andy > > > > > > > > > > > > > > > On Thu, 2006-08-03 at 09:44 -0700, Sunil Mushran wrote: > > > > 1. o2net talks tcp. It should be able to handle this. > > > > 2. If the cluster is active and the nodes are communicating, > > > > the keepalive packet is rarely sent. It only sends the packet > > > > if it does not hear from the other node for 5 secs. > > > > 3. Try the same with 1.2.3. (We made 2 important 1 line fixes.) > > > > 4. If this does happen again, and you are interested, we > > > > could always give you a drop that dumps the stack of > > > > all the procs, to get a better feel for the situation. > > > > > > > > Andy Phillips wrote: > > > > > Hello, > > > > > > > > > > Apologies for following up on myself. > > > > > > > > > > in ocfs2/cluster/tcp_internal.h > > > > > #define O2NET_KEEPALIVE_DELAY_SECS 5 > > > > > #define O2NET_IDLE_TIMEOUT_SECS 10 > > > > > > > > > > > > > > > Is this really sensible? Potentially, given small variance in > > > > > system clocks losing one keepalive packet (assuming that > > > > > o2net_sc_send_keep_req is the only thing keeping the connection alive) > > > > > the loss of one packet could cause a node to self fence and reboot. > > > > > > > > > > Would > > > > > #define O2NET_KEEPALIVE_DELAY_SECS 5 > > > > > #define O2NET_IDLE_TIMEOUT_SECS 20 > > > > > > > > > > Cause any problems? > > > > > > > > > > Andy > > > > > > > > > > > > > > > > > > > > On Thu, 2006-08-03 at 12:41 +0100, Andy Phillips wrote: > > > > > > > > > >> Hello, > > > > >> > > > > >> I've a two node 10gR2 rac cluster on a pair of sun opteron boxes. > > > > >> Redhat AS 4.3 2.6.9-34.0.1.ELsmp x86_64. ocfs 1.2.2. RAC is using > > > > >> ASM to talk to the data files, but we have 3 ocfs2 filesystems up > > > > >> to share dba files, and the usual bits and bobs. > > > > >> > > > > >> Things were fine until, on mostly idle system, this happened out > > > > >> of the blue; > > > > >> > > > > >> Aug 2 19:06:27 fred kernel: o2net: connection to node barney (num 0) > > at > > > > >> 172.16.6.10:7777 has been idle for 10 seconds, shutting it down. > > > > >> Aug 2 19:06:27 fred kernel: (0,7):o2net_idle_timer:1309 here are > > some > > > > >> times that might help debug the situation: (tmr 1154545576.798263 now > > > > >> 1154545586.796978 dr 1154545576.798238 adv > > > > >> 1154545576.798291:1154545576.798293 func (06aac8a1:1) > > > > >> 1154545566.800782:1154545566.800787) > > > > >> Aug 2 19:06:27 fred kernel: o2net: no longer connected to node > > barney > > > > >> (num 0) at 172.16.6.10:7777 > > > > >> Aug 2 19:08:33 fred kernel: (25,7):o2quo_make_decision:143 ERROR: > > > > >> fencing this node because it is connected to > > > > >> a half-quorum of 1 out of 2 nodes which doesn't include the lowest > > > > >> active node 0 > > > > >> Aug 2 19:08:33 fred kernel: (25,7):o2hb_stop_all_regions:1908 ERROR: > > > > >> stopping heartbeat on all active regions. > > > > >> > > > > >> And the node then halted. > > > > >> > > > > >> Barney is node 0. The systems were idle. We've hammered the ocfs2 > > > > >> file systems, and set o2cb_heartbeat_threshold to 61. All is good and > > > > >> stable under heavy i/o. > > > > >> > > > > >> The interconnect is a bonded interface, with two gig cards, each > > > > >> connected (with flow control on) to two separate FESX424 switches. > > > > >> The switches dont register any problems at this time, nor does linux > > > > >> register any interface issues. > > > > >> > > > > >> I'm looking at the source code at the moment, but nothing is > > leaping > > > > >> out at me. Any ideas - Do the timer debug lines above mean anything > > to > > > > >> anyone. > > > > >> > > > > >> Thanks > > > > >> Andy > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > > > > > ________________________________________________________________________ > > > > In order to protect our email recipients, Betfair use SkyScan from > > > > MessageLabs to scan all Incoming and Outgoing mail for viruses. > > > > > > > > ________________________________________________________________________ > > > -- > > > Andy Phillips, FRAS > > > Systems Architect, Information Systems. > > > Betfair.com > > > > > > Direct Line: 0208 834 8436 > > > > > > Betfair Limited (Company No.5140986), Winslow Road, Hammersmith > > > Embankment, London W6 9HP, United Kingdom, +44 208 834 8000, +44 208 834 > > > 8501 (direct). The information in this e-mail and any attachment is > > > confidential, may contain legal advice protected by privilege and is > > > intended only for the named recipient(s). The e-mail may not be > > > disclosed or used by any person other than the addressee, nor may it be > > > copied in any way. If you are not a named recipient please notify the > > > sender immediately and delete any copies of this message. Any > > > unauthorized copying, disclosure or distribution of the material in this > > > e-mail is strictly forbidden. Any view or opinions presented are solely > > > those of the author and do not necessarily represent those of the > > > company. > > > > > > _______________________________________________ > > > Ocfs2-users mailing list > > > [email protected] > > > http://oss.oracle.com/mailman/listinfo/ocfs2-users > > > > > > > > > ________________________________________________________________________ > > In order to protect our email recipients, Betfair use SkyScan from > > MessageLabs to scan all Incoming and Outgoing mail for viruses. > > > > ________________________________________________________________________ > -- > Andy Phillips, FRAS > Systems Architect, Information Systems. > Betfair.com > > Direct Line: 0208 834 8436 > > Betfair Limited (Company No.5140986), Winslow Road, Hammersmith > Embankment, London W6 9HP, United Kingdom, +44 208 834 8000, +44 208 834 > 8501 (direct). The information in this e-mail and any attachment is > confidential, may contain legal advice protected by privilege and is > intended only for the named recipient(s). The e-mail may not be > disclosed or used by any person other than the addressee, nor may it be > copied in any way. If you are not a named recipient please notify the > sender immediately and delete any copies of this message. Any > unauthorized copying, disclosure or distribution of the material in this > e-mail is strictly forbidden. Any view or opinions presented are solely > those of the author and do not necessarily represent those of the > company. > > _______________________________________________ > Ocfs2-users mailing list > [email protected] > http://oss.oracle.com/mailman/listinfo/ocfs2-users > _______________________________________________ Ocfs2-users mailing list [email protected] http://oss.oracle.com/mailman/listinfo/ocfs2-users
