Axton, you're 100% correct. And we thought of that too. We just don't know a way to ensure we hit _all_ of the open threads at least once an hour. BMC's suggestion was to "hit the server hard enough to use all the queues like you would under load testing" but I have the same doubts you do: will this cause deterioration in the user experience or server performance? I'm guessing, as you are, that it would.
I'm 99% sure the firewall is Cisco of some sort and you may be right on the state table only being created on SYN packets but that means that any SYN packet passing through the firewall (the start of any TCP connection) that passes a rule would be added to the state table. After that any traffic, regardless of packet type, would be covered by the entry in the state table as long as it was over the established connection, wouldn't it? Then the problem arises when the state table, to save firewall resources, clears out old, defunct connections. I'm glad someone else agrees that the best approach to this would be to eliminate the network devices that may be causing the issue rather than trying to engineer ARS to keep all the connections open. It does amaze me, though, that BMC can call ARS an "enterprise" product when it behaves so badly with stateful firewalls. J.T. Shyman Column Technologies Cell: 404-242-5407 -----Original Message----- From: Axton [mailto:[EMAIL PROTECTED] Sent: Wednesday, September 19, 2007 10:25 PM Subject: Re: ARS 7/Oracle and Firewalls/Network devices The escalation is (was) single threaded; in order to send traffic over every db connection, you have to exercise every thread. Since the escalation engine is single threaded, it will only occupy that one thread. If you notice in the arerror.log that all filter errors reported show 390693 as the rpc queue, it is executing everything on that one thread. In either case (single/multi-threaded escalation engine), it is only exercising the threads associated with the escalation engine, not the fast, list, callback, external auth, or custom queues. Axton Grams On 9/19/07, patrick zandi <[EMAIL PROTECTED]> wrote: > ** > Why not an afterhour escalation... instead.. > Say every 10 minutes.. to do table queries or a report or two.. > from 1800 - 0712 or something... > > > On 9/19/07, Axton <[EMAIL PROTECTED]> wrote: > > > > "Actually, now that I re-read your post I don't think putting a > > specific rule will side-step state checking." > > > > Depends on your firewall and the rule. Typically, states are created > > using only SYN packets, if state can be created on other packet types, > > you are still using stateful packet inspection, you are just allowing > > different packet types to add the session to the state table. > > > > "We talked to BMC a few weeks ago and they told us "theoretically" > > that it would be possible to write a custom API that would run custom > > workflow (neither of which they could give us) that would hit all of > > the server's Oracle connections at the same time often enough to > > prevent anything from seeing them as idle." > > > > I was thinking this as I was reading your email, though I am not sure > > how you would hit the admin and every fast/list/custom queue's threads > > without occupying all of them simultaneously. The api, to my > > knowledge, does not give you the capability to control what thread you > > are using, which means that your api will have to be multi-threaded > > and will have to occupy the max number of configured threads per rpc > > queue, which will cause your remedy server to appear to hang (i.e., > > block other operations on those queues). > > > > Can you share what type of firewall you are using? > > > > If you really want to remove the firewall from the equation, remove it > > from the network, or completely disable it. I can't see that vlan > > tagging would cause any issues with this. vlan's are configured in > > one of two way's, on the switch per port or the tagging is handled by > > the end nodes. If it is on the switch, it will be transparent to the > > client. > > > > Axton Grams > > > > On 9/19/07, J.T. Shyman < [EMAIL PROTECTED]> wrote: > > > ** > > > > > > > > > > > > Axton, > > > > > > > > > > > > Appreciate your input! > > > > > > > > > > > > I should have mentioned that we've been up and down that highway > and > > > > > > haven't seen a blasted thing. (apologies to Glen Frey) > > > > > > > > > > > > What you are saying is exactly what I thought and we've disabled > the > > > > > > idle timeout on the firewall. I know this may not be the same thing as > > > > > > preventing the firewall from using a state table but the firewall admin > > > > > > tells us he now sees idle connections with idle times > 60 minutes. So, > > > > > > we're kind of thinking we've eliminated the firewall as a > > > > > > cause...although we may not have, we aren't pursuing that any longer. > > > > > > > > > > > > Actually, now that I re-read your post I don't think putting a > > > > > > specific rule will side-step state checking. The purpose of a state > table > > > > > > on a firewall is to speed up handling of traffic by allowing already > known > > > > > > good traffic to pass without undergoing validation against the rulebase > > > > > > for every packet. Adding a rule that allows a single port connection, > > > > > > which is what we had before, doesn't stop the state table from > > > > > > functioning. In fact, it may actually be what causes the connection to > be > > > > > > put in the state table in the first place, no? Also, turning the > firewall > > > > > > into, effectively, a packet-based firewall might have a detrimental > affect > > > > > > on network throughput not only between AR and Oracle but for any other > > > > > > connections on that firewall due to increased overhead...or am I wrong? > > > > > > > > > > > > Additionally, we the firewall admin put in ANY<->ANY rule in place > a > > > > > > few nights ago and the problem is still occurring. I'd hoped that this > > > > > > would circumvent the state table but it apparently doesn't. > > > > > > > > > > > > I don't suppose there is a AR-based solution? We could keep trying > > > > > > changes on the network until we effectively distroy any semblance of the > > > > > > original network design but that wouldn't mollify anyone. In fact, it > may > > > > > > have the opposite affect. > > > > > > > > > > > > We talked to BMC a few weeks ago and they told us "theoretically" > > > > > > that it would be possible to write a custom API that would run custom > > > > > > workflow (neither of which they could give us) that would hit all of the > > > > > > server's Oracle connections at the same time often enough to prevent > > > > > > anything from seeing them as idle. Does this sound like a good approach > to > > > > > > anyone? Any and all thoughts and comments are welcome! > > > > > > > > > > > > Thanks! > > > > > > > > > > > > J.T. > > > > > > > > > > > > __20060125_______________________This posting was > > > submitted with HTML in it___ > > > > > ____________________________________________________________________________ ___ > > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where > the Answers Are" > > > > > > -- > Patrick Zandi __20060125_______________________This posting > was submitted with HTML in it___ ____________________________________________________________________________ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are"
