Thanks for your time, Axton. I spoke with our network engineer and he studied the firewall logs and sees quit a few denials and some resets, but is confident things should be set up correctly on the firewall. He only sees these denials/resets for the Midtier application, not some of our other apps.
Here is what we're observing on the Linux Midtier server. Below is an example of 3 existing tcp connections that remain intact indefinitely or until midtier is restarted. The firewall knows about these 3 tcp connections, but if the connections are idle for more than 60 minutes (default firewall setting), the firewall will drop these idle tcp connections. Then, when a user gets on midtier (after the idle period), midtier will attempt to use one or more of these tcp connections and the firewall responds with a Deny (since it has aged off the inactive connection). This is normal behavior for a firewall as I understand things, but the application (midtier in this case) should ideally make use of the tcp keepalive or inactivetly timeout supported by Linux. Apparently, midtier does not take advantage of this. I suspect other customers are having this problem IF their topology utilizes a firewall between Midtier and ARS and there are long periods of inactivity. The reason, however, they may not be complaining is because once you get the RCP failure (arerr 91), then subsequent connections seem to work fine. For the example below, once the arerr 91 timeout finally occurred, the associated tcp connection was terminated on the midtier server and a new one was created. $netstat -antopp|grep 2031 Proto Recv-Q Send-Q Local Address Foreign Address State Timer tcp 0 0 ::ffff:172.16.0.129:54310 ::ffff:10.1.1.141:2031 ESTABLISHED 24977/java off (0.00/0/0) tcp 0 864 ::ffff:172.16.0.129:54309 ::ffff:10.1.1.141:2031 ESTABLISHED 24977/java on (3.59/12/0) tcp 0 1416 ::ffff:172.16.0.129:43717 ::ffff:10.1.1.141:2031 ESTABLISHED 24977/java on (16.19/11/0) After a very long wait trying to login in, the user will receive this message: ARERR [91] RPC call failed : ONC/RPC call timed out Ken -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[email protected]] On Behalf Of Axton Sent: Tuesday, November 17, 2009 3:24 PM To: [email protected] Subject: Re: Firewall TCP Timeouts You need to configure your firewall properly. With a firewall, you can define what types of packets can create a state entry on the firewall. Typical is syn, syn ack. Even if there is no state entry, a new packet should create a new state (not be dropped). If a new packet does not create a new state, change the firewall rules so that it does. A network dump will tell you what type of packet is going out; the firewall logs should tell you what type of packet was rejected. I could see that this would be a problem if the midtier servers were behind a NAT. Is this the case? Axton Grams On Tue, Nov 17, 2009 at 12:07 PM, Leihkauff, Kenneth <[email protected]> wrote: > ** > > Hello, > > > > We have a firewall between our MidTier server and ARS system. The firewall > is configured to drop tcp connections after being idle for 60 minutes > (typical/default firewall setting). Several MidTier user sessions will make > use of a shared tcp connection so you might have 100 sessions but > significantly fewer tcp connections. During idle times (like at night), the > firewall will discard these idle tcp connections but the MidTier server will > still retain these tcp references (this can be seen by using "netstat > -anto"). So, when users get back on the system, MidTier apparently is > trying to utilize one of these defunct tcp connections so you end up with > problems like ARERR 91 rpc timeouts because these tcp connections are broken > pipes. > > > > Is there a MidTier/Tomcat or other setting that you have found addresses > this problem? > > > > Thanks. > > > > Ken > > > > Background: > > Version 7.5 patch 3 > > MidTier - Linux, Tomcat JSP > > ARS - Linux, Oracle 10g > > _Platinum Sponsor: [email protected] ARSlist: "Where the Answers > Are"_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are"
