option abortonclose in tcp mode valid?

[Reinhard Vicinus]

Hi,

we have recently activated multi threading in our haproxy (version
1.8.14) and stumbled over the following problem:

we balance ftp via haproxy and after enabling multi threading, trying to
transfer very small files via ftp sometimes failed, with log entries
like this:

 Oct 20 22:45:58 vptest02 haproxy[11753]: 127.0.0.1:56631
[20/Oct/2018:22:45:58.787] ftp ftp/ftp 1/-1/0 0 CC 2/2/1/1/0 0/0

I could narrow down the problem to the option abortonclose. If enabled
and the frontend receives the complete data transfer before fully
establishing a connection to the backend the backend connection gets
reseted and the line above is logged.

Attached is a test haproxy configuration were i was able to reproduce
the problem and a tcpdump showing the problem.

I'm not sure if this is a bug or the documentation of the abortonclose
option should be expanded. From the documentation I first thought that
the option abortonclose only works in conjunction with mode http,
because only the http protocol is mentioned. In tcp mode the question in
my opinion is: "When is a connection aborted?". Thinking about it I came
to the conclusion that the abortonclose can make sense on mode tcp too
as long as the protocol used on the connection requires an answer from
the server. But if that is the case then the documentation of
abortonclose should explicitly mentioning that there are tcp based
protocols with which it will produce errors because they don't expect an
answer from the server.

Regarding why it only happens with multi threading enabled: Could it be
that a single threaded haproxy always opens the backend connection fully
prior to closing the frontend connection and therefore avoided the
problem altogether?

But I'm no haproxy expert so I can be completely wrong and the issue is
something completely differently. Please let me know if further
information is needed to investigate this issue.

Regards
Reinhard Vicinus
global
cpu-map  auto:1/1-4 0-3
nbproc  1
nbthread  4

log /dev/loglocal0
log /dev/loglocal1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd 
listeners
stats timeout 30s
user haproxy
group haproxy
daemon

# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private

# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
#  
https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
ssl-default-bind-ciphers 
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3

defaults
log global
modehttp
option  httplog
option  dontlognull
option  abortonclose
timeout connect 5000
timeout client  5
timeout server  5
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend ftp
  bind 127.0.0.121:21
  bind 127.0.0.121:1-10250
  default_backend ftp
  mode tcp

backend ftp
  fullconn 128
  mode tcp
  server ftp 10.138.3.245 check port 21



abortonclose.pcap
Description: application/vnd.tcpdump.pcap

Re: High CPU Usage followed by segfault error

[Soji Antony]

Hi

FYI, following is the backtrace for segfault which we are seeing in the
kern.log.

Oct 18 10:11:30  kernel: [841364.001036] haproxy[30696]: segfault at 8 ip
5567eaf6aac2 sp 7ffdd70447b0 error 6 in haproxy[5567eae75000+172000]

# apport-retrace -g _usr_sbin_haproxy.0.crash
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from /usr/sbin/haproxy...Reading symbols from
/usr/lib/debug/.build-id/56/c5ffb3112d35c68a487caa1f4b788953891ade.debug...done.
done.
[New LWP 30696]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/var/run/haproxy.pid -sf 30646'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x5567eaf6aac2 in do_unbind_listener
(listener=listener@entry=0x5567ebd82a00,
do_close=do_close@entry=1) at src/listener.c:319
319 src/listener.c: No such file or directory.
(gdb) list
314 in src/listener.c
(gdb) backtrace
#0  0x5567eaf6aac2 in do_unbind_listener
(listener=listener@entry=0x5567ebd82a00,
do_close=do_close@entry=1) at src/listener.c:319
#1  0x5567eaf6b252 in unbind_listener
(listener=listener@entry=0x5567ebd82a00)
at src/listener.c:342
#2  0x5567eaf6b308 in unbind_all_listeners (proto=0x5567eb1fbcc0
) at src/listener.c:365
#3  0x5567eaf9bf0e in protocol_unbind_all () at src/protocol.c:76
#4  0x5567eaf397d6 in deinit () at src/haproxy.c:2289
#5  0x5567eaea1e1f in main (argc=, argv=)
at src/haproxy.c:3092

Haproxy 1.8.14 is installed from ubuntu repository

# cat vbernat-haproxy.list
deb http://ppa.launchpad.net/vbernat/haproxy-1.8/ubuntu trusty main
deb-src http://ppa.launchpad.net/vbernat/haproxy-1.8/ubuntu trusty main

Thanks

On Wed, Oct 17, 2018 at 9:26 AM Soji Antony  wrote:

> Hi Willy / Olivier,
>
> Thank you very much for the patch & detailed explanation. I will apply
> this patch on our servers.
>
> > Unfortunately, as is often the case with gdb, that's less than useful :/
> > If you have that available, you may install the haproxy-dbg package, but
> > I'm not convinced it will yield better results.
>
> PFA gdb.txt file which has 'thread apply all bt' & 'info threads' command
> output after installing haproxy-dbg package if that helps.
>
> > Can you share your config, obsucating any confidential informations, IP
> > addresses etc ?
>
> PFA haproxy.cfg file, I have added a comment [removed] wherever I have
> truncated lines.
>
> > You mentionned you where getting a segfault, do you know how to
> reproduce it ?
>
> Not sure how to reproduce it. I can see these segfault error messages in
> kernel logs on random servers.I can try enabling crash dump on one of the
> servers and share the details.
>
> Oct  9 16:16:35  kernel: [85669.521234] haproxy[59075]: segfault at
> 7fda1fb0fc60 ip 55c7273b643b sp 7fd8c2ffaab0 error 4 in
> haproxy[55c72734e000+172000]
> Oct 10 09:48:43  [148797.364018] haproxy[60048]: segfault at 8 ip
> 556ba5c7eac2 sp 7ffc5ef9e730 error 6 in haproxy[556ba5b89000+172000]
> Oct 11 14:30:56  kernel: [252130.055746] haproxy[4538]: segfault at
> 7fe088e87350 ip 5637ab43fea7 sp 7fe0857e8c20 error 4 in
> haproxy[5637ab41+172000]
> Oct 11 16:47:03 kernel: [260297.82] haproxy[74455]: segfault at
> 7f07d0de7290 ip 5574f96e1ea7 sp 7f07ce9c6c20 error 4 in
> haproxy[5574f96b2000+172000]
> Oct 11 22:06:19 : [279453.364729] haproxy[103724]: segfault at
> 7f7e492535d0 ip 55c8b4f1dea7 sp 7f7e46d93c20 error 4 in
> haproxy[55c8b4eee000+172000]
> Oct 13 04:31:14 : [388948.155673] haproxy[92338]: segfault at 8 ip
> 5583be079ac2 sp 7ffc6cb34e60 error 6 in haproxy[5583bdf84000+172000]
> Oct 15 15:17:04  kernel: [600498.581053] haproxy[63374]: segfault at 8 ip
> 55dd2e7d1ac2 sp 7ffed747e1d0 error 6 in haproxy[55dd2e6dc000+172000]
>
> > You also mentionned reloads are frequent, can you tell if the CPU spike
> happens
> immediately after a reload ?
>
> It is very difficult to say as the reloads are quite often. Attaching the
> graph for your reference. I can see that last reload happened at 16.56.19 &
> CPU spike usage started spiking at 16.57.30. But may be due to high CPU
> usage the script which we use to send this reload count to graphite might
> 

Re: HAproxy ssh connection closes fast , after logon

[Imam Toufique]

Hi Aleks,

Basically, I needed to bind to a different port then 22.  and , with
Brian's help, i ended up with the following config:

frontend www-ssh-proxy
  bind x.x.x.x:22
  mode tcp
  default_backend www-ssh-proxy-backend

backend www-ssh-proxy-backend
   mode tcp
   balance source
   server web1 10.1.100.156:22 check weight 1
   server web2 10.1.100.160:22 check weight 1

I have now changed the port 2200, to 22 -- by using 2200 for sshd itself
and 22 for binding.

This is not the perfect set up yet, but this is better then what I had
before.

thanks.

On Sat, Oct 20, 2018 at 1:39 AM Aleksandar Lazic  wrote:

> Hi.
>
> Am 20.10.2018 um 09:48 schrieb Imam Toufique:
> > Thanks, Jarno!
> >
> > Brian Gibson helped me out!  thanks, Brian!
>
> Would be nice to know what's the solution was, thanks.
>
> Regards
> Aleks
>
> > On Sat, Oct 20, 2018 at 12:10 AM Jarno Huuskonen  > > wrote:
> >
> > Hi,
> >
> > On Fri, Oct 19, Imam Toufique wrote:
> > > I am working on setting up ssh / sftp capability with HAProxy,
> > > initial setup is done ( thanks to some of the members in the
> haproxy email
> > > list for help! ) .  I ran into an issue  -- as soon as I ssh via
> the proxy
> > > node, within a minutes or so, ssh connection closes on me.
> >
> > Is the connection idle ? "in 50.3 seconds" matches your
> > timeout client/server 5(ms)
> > (Your (haproxy)logs should give more information why the connection
> > was closed).
> >
> > > here is my config file:
> > > -
> >
> > ...
> >
> > >timeout client 5
> > >timeout server 5
> >
> > ...
> >
> > > backend http_back
> > >timeout connect 90
> > >timeout server 90
> >
> > 90(ms)(=15minutes) connect timeout for http seems long ...
> > (AFAIK this is how long haproxy will wait for tcp connection
> > to a backend http server).
> >
> > > backend www-ssh-proxy-backend
> > >mode tcp
> > >balance roundrobin
> > >stick-table type ip size 200k expire 30m
> > >stick on src
> > >default-server inter 1s
> > >server web1 10.1.100.156:22  check id 1
> > >server web2 10.1.100.160:22  check id 2
> >
> > Try using longer timeout server on www-ssh-proxy-backend.
> > (and/or longer timeout client on www-ssh-proxy).
> >
> > (You could also try to play with sshd_config: ClientAliveInterval and
> > TCPKeepAlive)
> >
> > > Transferred: sent 3312, received 3184 bytes, in 50.3 seconds
> >
> > -Jarno
> >
> > --
> > Jarno Huuskonen
> >
> >
> >
> > --
> > Regards,
> > */Imam Toufique/*
> > /*213-700-5485*/
>
>

-- 
Regards,
*Imam Toufique*
*213-700-5485*

Re: HAproxy ssh connection closes fast , after logon

[Aleksandar Lazic]

Hi.

Am 20.10.2018 um 09:48 schrieb Imam Toufique:
> Thanks, Jarno!
> 
> Brian Gibson helped me out!  thanks, Brian!

Would be nice to know what's the solution was, thanks.

Regards
Aleks

> On Sat, Oct 20, 2018 at 12:10 AM Jarno Huuskonen  > wrote:
> 
> Hi,
> 
> On Fri, Oct 19, Imam Toufique wrote:
> > I am working on setting up ssh / sftp capability with HAProxy,
> > initial setup is done ( thanks to some of the members in the haproxy 
> email
> > list for help! ) .  I ran into an issue  -- as soon as I ssh via the 
> proxy
> > node, within a minutes or so, ssh connection closes on me.
> 
> Is the connection idle ? "in 50.3 seconds" matches your
> timeout client/server 5(ms)
> (Your (haproxy)logs should give more information why the connection
> was closed).
> 
> > here is my config file:
> > -
> 
> ...
> 
> >    timeout client 5
> >    timeout server 5
> 
> ...
> 
> > backend http_back
> >    timeout connect 90
> >    timeout server 90
> 
> 90(ms)(=15minutes) connect timeout for http seems long ...
> (AFAIK this is how long haproxy will wait for tcp connection
> to a backend http server).
> 
> > backend www-ssh-proxy-backend
> >    mode tcp
> >    balance roundrobin
> >    stick-table type ip size 200k expire 30m
> >    stick on src
> >    default-server inter 1s
> >    server web1 10.1.100.156:22  check id 1
> >    server web2 10.1.100.160:22  check id 2
> 
> Try using longer timeout server on www-ssh-proxy-backend.
> (and/or longer timeout client on www-ssh-proxy).
> 
> (You could also try to play with sshd_config: ClientAliveInterval and
> TCPKeepAlive)
> 
> > Transferred: sent 3312, received 3184 bytes, in 50.3 seconds
> 
> -Jarno
> 
> -- 
> Jarno Huuskonen
> 
> 
> 
> -- 
> Regards,
> */Imam Toufique/*
> /*213-700-5485*/

Re: HAproxy ssh connection closes fast , after logon

[Imam Toufique]

Thanks, Jarno!

Brian Gibson helped me out!  thanks, Brian!

On Sat, Oct 20, 2018 at 12:10 AM Jarno Huuskonen 
wrote:

> Hi,
>
> On Fri, Oct 19, Imam Toufique wrote:
> > I am working on setting up ssh / sftp capability with HAProxy,
> > initial setup is done ( thanks to some of the members in the haproxy
> email
> > list for help! ) .  I ran into an issue  -- as soon as I ssh via the
> proxy
> > node, within a minutes or so, ssh connection closes on me.
>
> Is the connection idle ? "in 50.3 seconds" matches your
> timeout client/server 5(ms)
> (Your (haproxy)logs should give more information why the connection
> was closed).
>
> > here is my config file:
> > -
>
> ...
>
> >timeout client 5
> >timeout server 5
>
> ...
>
> > backend http_back
> >timeout connect 90
> >timeout server 90
>
> 90(ms)(=15minutes) connect timeout for http seems long ...
> (AFAIK this is how long haproxy will wait for tcp connection
> to a backend http server).
>
> > backend www-ssh-proxy-backend
> >mode tcp
> >balance roundrobin
> >stick-table type ip size 200k expire 30m
> >stick on src
> >default-server inter 1s
> >server web1 10.1.100.156:22 check id 1
> >server web2 10.1.100.160:22 check id 2
>
> Try using longer timeout server on www-ssh-proxy-backend.
> (and/or longer timeout client on www-ssh-proxy).
>
> (You could also try to play with sshd_config: ClientAliveInterval and
> TCPKeepAlive)
>
> > Transferred: sent 3312, received 3184 bytes, in 50.3 seconds
>
> -Jarno
>
> --
> Jarno Huuskonen
>


-- 
Regards,
*Imam Toufique*
*213-700-5485*

Re: HAproxy ssh connection closes fast , after logon

[Jarno Huuskonen]

Hi,

On Fri, Oct 19, Imam Toufique wrote:
> I am working on setting up ssh / sftp capability with HAProxy,
> initial setup is done ( thanks to some of the members in the haproxy email
> list for help! ) .  I ran into an issue  -- as soon as I ssh via the proxy
> node, within a minutes or so, ssh connection closes on me.

Is the connection idle ? "in 50.3 seconds" matches your
timeout client/server 5(ms)
(Your (haproxy)logs should give more information why the connection
was closed).

> here is my config file:
> -

...

>timeout client 5
>timeout server 5

...

> backend http_back
>timeout connect 90
>timeout server 90

90(ms)(=15minutes) connect timeout for http seems long ...
(AFAIK this is how long haproxy will wait for tcp connection
to a backend http server).

> backend www-ssh-proxy-backend
>mode tcp
>balance roundrobin
>stick-table type ip size 200k expire 30m
>stick on src
>default-server inter 1s
>server web1 10.1.100.156:22 check id 1
>server web2 10.1.100.160:22 check id 2

Try using longer timeout server on www-ssh-proxy-backend.
(and/or longer timeout client on www-ssh-proxy).

(You could also try to play with sshd_config: ClientAliveInterval and
TCPKeepAlive)

> Transferred: sent 3312, received 3184 bytes, in 50.3 seconds

-Jarno

-- 
Jarno Huuskonen