FYI in case someone is facing the same problem:
I opened a case at foundry as this was replicate able on all apache
mod_proxy or rewrite setups on Solaris as well as on Linux I tested.
If you have some slower apps it looks like apache as a client is
handling proxy requests slower in terms of TCP closing - the last
(apache) client FIN is delayed. I saw this with some java clients, too.
I got great support from the new TAC in Amsterdam and we jointly setup
a test system where the TAC was able to replicate the issue.
Looks like there are two age timers: one for the forwarding session
and one for the reverse session which are not synced by default.
Because of that the delayed client FIN was forwarded to the correct
real server but no nat was done for the delayed packet by the SI as
the session has timed out already.
I got the hint by the TAC engineer to configure:
server no-one-way-session-age
That fixed all our issues.
Nils
On Sep 7, 2008, at 2:43 PM, Nils Domrose wrote:
Nils Domrose wrote:
Hi,
while we introduced a new frontend firewall between out Frontend
Apache server and out Backends, I experienced a strange behavior:
in front of out Backend Servers we have a pair of SI4G running
10.2.0 Software and source-nat.
Frontend Server=Client
Backend Server=Real Server
On the Firewall I can see frequent blocks of FIN ACK packets
(Client to VIP) and RST Packets (Real! Server to Client).
Of cause I see the FIN ACK blocks only after the tcp.closing timer
on the firewall has expired due to the retransmits. I do see the
blocked RST packets all the time since there is no state and the
firewall does not know anything about the real server the RST is
coming from.
After running multiple snoops the picture looks like this:
1. Client establishes a connection to the VIP (HTTP 1.1).
2. Multiple requests are served via the connection.
3. After a while the connection is idle (and all Packets are ACK'ed
4. After 15 seconds idle as configured on apache side, the backend
server sends a FIN (ACK) to close the connection
5. The Frontend server sends immediately an ACK
now the interesting part starts:
6. after 80+ seconds the Frontend Server send the FIN ACK
7. the Packet is not NAT'ed any more when reaching the real server
(Source: Frontend Server -> Dest: real server).
also due to the missing NAT the port number does no longer match
the port number of the previous connection from real server point
of view.
(the previous connection was source-nat IP:nated port -> real
server:9080
8. The real server sends a RST (due to the unknown TCP connection)
to the Frontend Server
Of cause 6 to 8 happen several times afterwards since the frontend
server never receives an answer for the FIN ACK and retransmits the
FIN ACK.
I wonder why apache as a client (sometimes) needs more than 80
seconds to send a final FIN ACK but i have not found any
information that this is wrong/ not allowed. We run Apache 2.0
using mod rewrite (we also tested mod proxy) in frontend and backend.
Does anyone have an idea what timeout value is responsible for the
deletion of the Serveriron session and is causing the source NAT to
fail?
When does this timeout start (i guess once the Serveriron recieves
the first FIN)
And last but not least is this configurable ?
Any hint is welcome!
Nils
_______________________________________________
foundry-nsp mailing list
[email protected]
http://puck.nether.net/mailman/listinfo/foundry-nsp
_______________________________________________
foundry-nsp mailing list
[email protected]
http://puck.nether.net/mailman/listinfo/foundry-nsp
_______________________________________________
foundry-nsp mailing list
[email protected]
http://puck.nether.net/mailman/listinfo/foundry-nsp
