Re: tcp mode: client port selection
On Fri, Mar 03, 2023 at 04:04:37PM +0100, Amaury Denoyelle wrote:
> > Can anyone say sth. about client port allocation in haproxy? Is it done
> > manually in some cases? Or is that a task that is completely done by the OS?
>
> To my knowledge, haproxy does not explicitely select the port when
> connecting to a backend server unless a specific "source" statement is
> used, so this should be the responsibility of the OS. Have you checked
> that your ephemeral port range is big enough ?
>
> $ sysctl net.ipv4.ip_local_port_range net.ipv4.ip_local_reserved_ports
Indeed, source ports are chosen by the operating system. The problem
Jack reports is a well-known issue in NAT+multi-homed environments and
is purely an architectural one:
client 1.2.3.4:5678
|
|
/\
{ internet }
`'
/ \
/ \
ISP A ISP B
| |
(X) (X)
\ /
2.2.2.2:443 \ / 3.3.3.3:443
+-+
| DNAT|
+-+
|
server4.4.4.4:443
The DNAT box that receives traffic from both ISPs needs to store
a session in its table that contains this:
ext src,dstint src,dst
1.2.3.4:5678 2.2.2.2:443 1.2.3.4:5678 4.4.4.4:443
This way when the server responds to 1.2.3.4:5678, the NAT box reads this
NAT table in reverse direction and finds that 4.4.4.4:443->1.2.3.4:5678
must be translated to 2.2.2.2:443->1.2.3.4:5678.
But what if the same source arrives on the other public IP ? you'll get
a new entry:
ext src,dstint src,dst
1.2.3.4:5678 3.3.3.3:443 1.2.3.4:5678 4.4.4.4:443
and suddenly your NAT table contains a conflict for the reverse lookup
needed for responses:
ext src,dstint src,dst
1.2.3.4:5678 2.2.2.2:443 1.2.3.4:5678 4.4.4.4:443
1.2.3.4:5678 3.3.3.3:443 1.2.3.4:5678 4.4.4.4:443
which one to pick for 4.4.4.4:443->1.2.3.4:5678 ?
In order to avoid this, some NAT boxes support to implement a
double-nat mechanism for incoming traffic. They'll either set a
specific source per original destination, or will assign different
port ranges, so you could have this:
ext src,dstint src,dst
1.2.3.4:5678 2.2.2.2:443 4.4.4.2:5678 4.4.4.4:443
1.2.3.4:5678 3.3.3.3:443 4.4.4.3:5678 4.4.4.4:443
The problem is that the server doesn't receive the original IPs
anymore, which can be a problem for filtering or even legal
logging. In this case you'll need to rely on the NAT box's
logging, or it should implement a mechanism to pass the source
address via an auxiliary mechanism such as the PROXY protocol.
A much simpler method for the device is in fact to assign two
IP addresses to the server and configure the DNAT box to use a
distinct IP address for each ISP:
client 1.2.3.4:5678
|
|
/\
{ internet }
`'
/ \
/ \
ISP A ISP B
| |
(X) (X)
\ /
2.2.2.2:443 \ / 3.3.3.3:443
+-+
| DNAT|
+-+
| |
4.4.4.4:443 | | 4.4.4.5:443
server
This way there is no more ambiguity in the DNAT table:
ext src,dstint src,dst
1.2.3.4:5678 2.2.2.2:443 1.2.3.4:5678 4.4.4.4:443
1.2.3.4:5678 3.3.3.3:443 1.2.3.4:5678 4.4.4.5:443
If the server responds from 4.4.4.4 then the outgoing source
is necessarily 2.2.2.2 but if it responds from 4.4.4.5, then
the outgoing source is necessarily 3.3.3.3.
If for any reason you cannot fix this deployment this way and
can only act on the haproxy side (e.g. the server is at a
customer's who doesn't want to fix their broken setup and who
decided that it was your problem only), then as Amaury suggests,
the "source" directive allows you to enforce source port ranges
for outgoing traffic. Then you can just have two sets of non-
overlapping ports for each target address:
server isp-a 2.2.2.2:443 source 0.0.0.0:2-2
server isp-b 3.3.3.3:443 source 0.0.0.0:3-3
That way there will never be any conflict in the fortigate's table
and the return traffic will always find a unique path.
But if the setup is yours, I strongly encourage you to fix it as
explained above because any other client could face this problem.
And in some mobile environments which make use of CGN (carrier-grade
NAT), it is possible that a mobile client will use a very narrow
source port range and will frequently reuse the same ports, which
can cause exactly this problem to happen when connecting to that
site.
Hoping th
Re: tcp mode: client port selection
On Fri, Mar 03, 2023 at 09:35:45AM +0100, Jack Bauer wrote: > Am Do., 2. März 2023 um 17:52 Uhr schrieb Amaury Denoyelle < > adenoye...@haproxy.com>: > > > > It seems you do not use 'option redispatch' in your configuration so a > > retry will never be conducted on another server. Therefore, your problem > > is probably not related to haproxy retries. > > > From the documentation ( > http://docs.haproxy.org/2.7/configuration.html#4-option%20redispatch) one > could or should conclude, that option redispatch is only working in HTTP > mode. I confirm that it works also for proxy on TCP mode and that the documentation is confusing. > Even if it is also working in TCP mode and we are not using it in the > configuration, haproxy makes connections with the same client ip port to > another target server. > Can anyone say sth. about client port allocation in haproxy? Is it done > manually in some cases? Or is that a task that is completely done by the OS? To my knowledge, haproxy does not explicitely select the port when connecting to a backend server unless a specific "source" statement is used, so this should be the responsibility of the OS. Have you checked that your ephemeral port range is big enough ? $ sysctl net.ipv4.ip_local_port_range net.ipv4.ip_local_reserved_ports -- Amaury Denoyelle
Re: tcp mode: client port selection
Am Do., 2. März 2023 um 17:52 Uhr schrieb Amaury Denoyelle < adenoye...@haproxy.com>: > > It seems you do not use 'option redispatch' in your configuration so a > retry will never be conducted on another server. Therefore, your problem > is probably not related to haproxy retries. > >From the documentation ( http://docs.haproxy.org/2.7/configuration.html#4-option%20redispatch) one could or should conclude, that option redispatch is only working in HTTP mode. Even if it is also working in TCP mode and we are not using it in the configuration, haproxy makes connections with the same client ip port to another target server. Can anyone say sth. about client port allocation in haproxy? Is it done manually in some cases? Or is that a task that is completely done by the OS? -Jack
Re: tcp mode: client port selection
On Thu, Mar 02, 2023 at 02:39:25PM +0100, Jack Bauer wrote: > Hi, > [...] > Therefore it might be necessary to understand how haproxy chooses the > client-ip-port when creating new tcp connections to backend servers. > Especially how haproxy behaves when a connection attempt fails and a retry > is triggered because this retry might change the backend server. > Will the retry use the same client port? If yes, is this intended? > It seems you do not use 'option redispatch' in your configuration so a retry will never be conducted on another server. Therefore, your problem is probably not related to haproxy retries. -- Amaury Denoyelle
tcp mode: client port selection
Hi, I have a question regarding source port selection when using haproxy in tcp mode. We are using haproxy with the following scenario: | public haproxy with tcpmode | -> | PNAT Fortigate FW | -> | internal Host | The Fortigate is reachable via two ISPs (ISP-A, ISP-B) and has port forwards (NAT) to an internal host. The haproxy backend configuration contains both public IPs assigned to the Fortigate FW, enabling a distribution of the connections on both ISP connections (haproxy config appended). In this scenario we are getting "session clash" messages on the Fortigate FW. A session clash message indicates, that the tuple of for tcp connection tracking (sessions in the Fortigate world) is reused, before the previous session was closed. >From the Fortigate knowledgebase, the tupel is calculated from [HAPROXY-IP]:49660 and [INTERNAL-HOST-IP]:443 (Ports used from the example message appended, where 49660 is the haproxy client ip port) The message shows, that haproxy created a session / tcp connection using the ISP-B-IP and then created a new connection using the ISP-A-IP using the same client-ip-port as in the previous connection. This leads to unexpected behaviour (e.g. packet drop) and I want to get rid of these messages (and behaviour). Therefore it might be necessary to understand how haproxy chooses the client-ip-port when creating new tcp connections to backend servers. Especially how haproxy behaves when a connection attempt fails and a retry is triggered because this retry might change the backend server. Will the retry use the same client port? If yes, is this intended? Best regards, Jack Additional infos: === - Connection / session creation rate is low at around 3/s with peaks of 25/s. - connection duration is low - peak total concurrent connections is around 600 haproxy.cfg: == global maxconn 1 log stdout format raw daemon notice defaults log global mode tcp retries 2 timeout client 120s timeout connect 4s timeout server 120s timeout check 5s option tcpka option tcplog option logasap balance leastconn frontend tcp_443_frontend_with_proxy_protocol mode tcp bind *:443 accept-proxy maxconn $MAX_CONN default_backend tcp_443_backend backend tcp_443_backend mode tcp fullconn $MAX_CONN option tcp-check default-server init-addr libc,last,none server ip1_port1 [ISP-A-IP]:443 send-proxy maxconn $MAX_CONN check inter 1s server ip2_port1 [ISP-B-IP]:443 send-proxy maxconn $MAX_CONN check inter 1s Example message (the actual IP addresses have been replaced by placeholders): === New Status [HAPROXY-IP]:49660->[ISP-A-IP]:443([INTERNAL-HOST-IP]:443) dir=0 act=1 hook=4 [HAPROXY-IP]:49660->[INTERNAL-HOST-IP]:443([INTERNAL-FW-IP]:22686) dir=1 act=2 hook=0 [INTERNAL-HOST-IP]:443->[INTERNAL-FW-IP]:22686([HAPROXY-IP]:49660) dir=1 act=1 hook=4 [INTERNAL-HOST-IP]:443->[HAPROXY-IP]:49660([ISP-A-IP]:443) Old Status [HAPROXY-IP]:49660->[ISP-B-IP]:443([INTERNAL-HOST-IP]:443) dir=0 act=1 hook=4 [HAPROXY-IP]:49660->[INTERNAL-HOST-IP]:443([INTERNAL-FW-IP]:49660) dir=1 act=2 hook=0 [INTERNAL-HOST-IP]:443->[INTERNAL-FW-IP]:49660([HAPROXY-IP]:49660) dir=1 act=1 hook=4 [INTERNAL-HOST-IP]:443->[HAPROXY-IP]:49660([ISP-B-IP]:443)
