2009/9/5 Willy Tarreau <[email protected]>:
>
> Among the things I could think of, one of them would be the
> local system sometimes running out of source ports. You could
> check this as well as the fact that the system allows you to
> reuse connections still in TIME_WAIT state :
>
> /proc/sys/net/ipv4/ip_local_port_range
Gives a fairly large range:
32768 61000
> /proc/sys/net/ipv4/tcp_tw_reuse
Gives:
0
"netstat -ntp | wc -l" gives:
4397
"netstat -ntp | grep -v TIME_WAIT | wc -l" gives:
445
I don't think there are too many active connections, as the port range
allows for 28k, and I have 4k. Reusing TIME_WAIT connections may not
be necessary.
> Also, please check with tcpdump if you see duplicate SYNs. This
> is important, because a duplicate SYN will indicate that a port
> could be allocated but the connection could not be establish for
> whatever reason. However, the lack of duplicate SYNs would rather
> indicate a source port shortage.
This is what I did to check duplicate SYN:
In window 1, to detect haproxy retries:
tail -f haproxy.log | egrep ' [0-9]+/[0-9]+/[0-9]+/[0-9]+/[1-9]
[0-9]+/[0-9]+ '
In window 2:
tcpdump -w ~/tcp.log -Svvn 'port 83 and host s2.site.com'
I stop tcpdump with Ctrl-C as soon I notice in window 1 that a new
line has been reported:
Sep 5 22:15:55 s1 haproxy[3517]: 127.0.0.1:45818
[05/Sep/2009:22:15:50.708] apache/s2 0/0/5003/0/5003 200 1449 - - ----
0/0/0/0/1 0/0 {data.site.com} "GET /img/pinsets/1000/10200-s.png
HTTP/1.1"
I loaded the pcap file in wireshark and noticed two things:
1) the source port is not the same as logged in the haproxy logs, it
may be useful to have both the source port connecting to haproxy, and
the source port connecting to the backend.
2) Using this filter: "tcp.dstport==83 and tcp.flags.syn==1" I iterate
over all lines and notice that the source port is never the same, so
it may imply that there is no duplicate SYN.
Best regards,
--
Jean-Baptiste Quenot
http://jbq.caraldi.com/
