Deny with 413 request too large

[Joao Morais]

Hello list, I need to configure HAProxy to deny the request with 413 based on 
the value of the content-length header. This is my actual configuration:

errorfile 400 /usr/local/etc/haproxy/errors/413.http
http-request deny deny_status 400 if { req.body_size gt 10485760 }

This is working but sounds a hacky workaround since I’m using another status 
code. If I try to use:

errorfile 413 /usr/local/etc/haproxy/errors/413.http
http-request deny deny_status 413 if { req.body_size gt 10485760 }

... HAProxy complains with:

[WARNING] 135/001448 (27) : parsing [/etc/haproxy/haproxy.cfg:15] : status 
code 413 not handled by 'errorfile', error customization will be ignored.
[WARNING] 135/001448 (27) : parsing [/etc/haproxy/haproxy.cfg:89] : status 
code 413 not handled, using default code 403.

How should I configure HAProxy in order to deny with 413?

~jm

HAProxy Page Element Request Times Slowly Increase

[Caleb Anthony]

Hello All,

I've got a strange issue with our deployment of HAProxy 1.6 where a user
will make a request to a page, and as each page element comes in, the time
taken shown on each file downloaded in the IE developer tools increases by
a factor of 3, until finally the user is prompted for credentials, but the
page load never continues. I should mention that this is an ASP.NET web
application running with Integrated Windows Authentication on IIS 8. I
know, I know, IE + IIS + IWA = terrible, but it's hard for the enterprise
to not use these platforms internally, especially the IWA single sign on
part.

So a request looks like this in the developer tools:

mainpage.aspx - HTTP 200 - 130.04KB - 421ms
css file - HTTP 200 - 1.51KB - 296ms
css file - HTTP 200 - 14.93KB - 0.96s
css file - HTTP 200 - 13.99KB - 2.73s
css file - HTTP 200 - 29.92KB - 8.14s
css file - HTTP 200 - 23.44KB - 24.36s
webresource.axd - Pending - Pending - Pending (this is where the user is
prompted for credentials)

And on that last file is where it then prompts for credentials, which never
works because I believe that HAProxy has hit the timeout and closed the
connection, and breaking IWA.

I should mention that our HAProxy server works great 99.9% of the time, and
we do a lot of L7 load balancing on different URLs, and host headers, and
we handle a good amount of traffic that returns page elements in ms, and it
also handles Integrated Windows Auth fine - most of the time. It's just
these occasional things that have me at a loss.

Any suggestions from the mailing list?

Re: haproxy "inter" and "timeout check", retries and "fall"

[Bryan Talbot]

> On May 13, 2017, at May 13, 10:59 PM, Jiafan Zhou  
> wrote:
> 
> 
> Hi all,
> 
> The version of haproxy I use is: 
> 
> # haproxy -version
> HA-Proxy version 1.5.2 2014/07/12
> Copyright 2000-2014 Willy Tarreau  

This version is so old. I’m sure there must be hundreds of bugs fixed over the 
last 3 years. Why not use a properly current version?


> I have a question regarding the Health Check. In the documentation of 
> haproxy, it mentions the below for the "timeout check" and "inter":
> 
> Now I am wondering here which one and what value will be used for healthcheck 
> interval. Is it "timeout check" as 10 seconds, or the "inter" as the default 
> 2 seconds?
> 
> 

Why not just set the health check values that you care about and not worry 
about guessing what they’ll end up being when only some are set and some are 
using defaults? If you need / expect them to be a particular value for proper 
system operation, I’d set them no matter what the defaults may be declared to 
be. 


> Another question, since I defined the "retries" to be 3, in the case of 
> server connection failure, will it reconnect 3 times? Or does it use the 
> "fall" parameter (which defaults to 3 here as well) instead for healthcheck 
> retry?
> 
> 


“retries” is for dispatching requests and is not used for health checks.


> So in this configuration, in the case of server failure, does it wait for up 
> to 30 seconds (3 fall or retries), then 20 seconds (2 rise), before the 
> server is considered operational? (in total 50 seconds)
> 
> 

retries are not considered, only health check specific settings like “fail”, 
“inter"

> Thanks,
> 
> Jiafan
> 

Re: OpenSSL engine and async support

[Grant Zhang]

> On May 15, 2017, at 03:14, Emeric Brun  wrote:
> 
> What does it look like?
New patches attached.

> 
> The issue is very similar: 
> https://mta.openssl.org/pipermail/openssl-dev/2016-March/005909.html
Interesting. yeah, it looks similar.

Regards,

Grant



0001-ssl-add-basic-support-for-OpenSSL-crypto-engine.patch
Description: Binary data


0002-ssl-add-openssl-async-mode-support.patch
Description: Binary data

Re: OpenSSL engine and async support

[Emeric Brun]

Hi Grant,

On 05/15/2017 12:14 PM, Emeric Brun wrote:
> On 05/13/2017 01:14 AM, Grant Zhang wrote:
>>
>>> On May 10, 2017, at 04:51, Emeric Brun  wrote:
>>>
 It looks like the main process stalls at DH_free(local_dh_1024) (part of 
 __ssl_sock_deinit). Not sure why but I will debug and report back.

 Thanks,
>>>
>>> I experienced the same issue (stalled on a futex) if i run haproxy in 
>>> foreground and trying to kill it with kill -USR1.
>>>
>>> With this conf (dh param and ssl-async are disabled)
>>> global
>>> #   tune.ssl.default-dh-param 2048
>>>ssl-engine qat
>>> #   ssl-async
>>>nbproc 1
>>
>> It looks like that the stall on futex issue is related to DH_free() calling 
>> ENGINE_finish in openssl 1.1:
>> https://github.com/openssl/openssl/blob/master/crypto/dh/dh_lib.c#L109
>> (gdb) bt
>> #0  __lll_lock_wait () at 
>> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
>> #1  0x7fa1582c5571 in pthread_rwlock_wrlock ()
>> at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S:118
>> #2  0x7fa158a58559 in CRYPTO_THREAD_write_lock () from 
>> /tmp/openssl_1.1.0_install/lib/libcrypto.so.1.1
>> #3  0x7fa1589d8800 in ENGINE_finish () from 
>> /tmp/openssl_1.1.0_install/lib/libcrypto.so.1.1
>> #4  0x7fa158975e76 in DH_free () from 
>> /tmp/openssl_1.1.0_install/lib/libcrypto.so.1.1
>> #5  0x00417c78 in free_dh () at src/ssl_sock.c:7905
>> #6  0x7fa1591d58ce in _dl_fini () at dl-fini.c:254
>> #7  0x7fa158512511 in __run_exit_handlers (status=0, 
>> listp=0x7fa15888e688, run_list_atexit=true)
>> at exit.c:78
>> #8  0x7fa158512595 in __GI_exit (status=) at exit.c:100
>> #9  0x00408814 in main (argc=4, argv=0x7ffe72188548) at 
>> src/haproxy.c:2235
>>
>> Openssl 1.1 has changed the way ENGINE_cleanup works:
>> https://www.openssl.org/docs/man1.1.0/crypto/ENGINE_cleanup.html
>> "From OpenSSL 1.1.0 it is no longer necessary to explicitly call 
>> ENGINE_cleanup and this function is deprecated. Cleanup automatically takes 
>> place at program exit."
>>
>> I suppose by the time the destructor __ssl_sock_deinit is called, 
>> engine-related cleanup are already done by openssl and ENGINE_finish (from 
>> DH_free) stalls on a non-existing write lock.
>>
>> I have a workaround which moves the DH_free logic out of the destructor 
>> __ssl_sock_deinit, and right before process exit.  With the workaround I no 
>> longer see the stall issue. I am not sure whether it is optimal solution 
>> though. Let me know.
> 
> What does it look like?
> 
> The issue is very similar: 
> https://mta.openssl.org/pipermail/openssl-dev/2016-March/005909.html
> 
> 
>>
>> Thanks,
>>
>> Grant
> R,
> Emeric
> 
> 


Reading the changelog openssl-1.1.0e changelog:

  *) Make various cleanup routines no-ops and mark them as deprecated. Most
 global cleanup functions are no longer required because they are handled
 via auto-deinit (see OPENSSL_init_crypto and OPENSSL_init_ssl man pages).
 Explicitly de-initing can cause problems (e.g. where a library that uses
 OpenSSL de-inits, but an application is still using it). The affected
 functions are CONF_modules_free(), ENGINE_cleanup(), OBJ_cleanup(),
 EVP_cleanup(), BIO_sock_cleanup(), CRYPTO_cleanup_all_ex_data(),
 RAND_cleanup(), SSL_COMP_free_compression_methods(), ERR_free_strings() and
 COMP_zlib_cleanup().
 [Matt Caswell

and reading the OPENSSL_init_ssl/OPENSSL_init_crypto man pages.


We may remove all openssl uninit calls from the ssl_sock.c destructor for 
openssl v >= 1.1.

R,
Emeric
  

Re: OpenSSL engine and async support

[Emeric Brun]

On 05/13/2017 01:14 AM, Grant Zhang wrote:
> 
>> On May 10, 2017, at 04:51, Emeric Brun  wrote:
>>
>>> It looks like the main process stalls at DH_free(local_dh_1024) (part of 
>>> __ssl_sock_deinit). Not sure why but I will debug and report back.
>>>
>>> Thanks,
>>
>> I experienced the same issue (stalled on a futex) if i run haproxy in 
>> foreground and trying to kill it with kill -USR1.
>>
>> With this conf (dh param and ssl-async are disabled)
>> global
>> #   tune.ssl.default-dh-param 2048
>>ssl-engine qat
>> #   ssl-async
>>nbproc 1
> 
> It looks like that the stall on futex issue is related to DH_free() calling 
> ENGINE_finish in openssl 1.1:
> https://github.com/openssl/openssl/blob/master/crypto/dh/dh_lib.c#L109
> (gdb) bt
> #0  __lll_lock_wait () at 
> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
> #1  0x7fa1582c5571 in pthread_rwlock_wrlock ()
> at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S:118
> #2  0x7fa158a58559 in CRYPTO_THREAD_write_lock () from 
> /tmp/openssl_1.1.0_install/lib/libcrypto.so.1.1
> #3  0x7fa1589d8800 in ENGINE_finish () from 
> /tmp/openssl_1.1.0_install/lib/libcrypto.so.1.1
> #4  0x7fa158975e76 in DH_free () from 
> /tmp/openssl_1.1.0_install/lib/libcrypto.so.1.1
> #5  0x00417c78 in free_dh () at src/ssl_sock.c:7905
> #6  0x7fa1591d58ce in _dl_fini () at dl-fini.c:254
> #7  0x7fa158512511 in __run_exit_handlers (status=0, 
> listp=0x7fa15888e688, run_list_atexit=true)
> at exit.c:78
> #8  0x7fa158512595 in __GI_exit (status=) at exit.c:100
> #9  0x00408814 in main (argc=4, argv=0x7ffe72188548) at 
> src/haproxy.c:2235
> 
> Openssl 1.1 has changed the way ENGINE_cleanup works:
> https://www.openssl.org/docs/man1.1.0/crypto/ENGINE_cleanup.html
> "From OpenSSL 1.1.0 it is no longer necessary to explicitly call 
> ENGINE_cleanup and this function is deprecated. Cleanup automatically takes 
> place at program exit."
> 
> I suppose by the time the destructor __ssl_sock_deinit is called, 
> engine-related cleanup are already done by openssl and ENGINE_finish (from 
> DH_free) stalls on a non-existing write lock.
> 
> I have a workaround which moves the DH_free logic out of the destructor 
> __ssl_sock_deinit, and right before process exit.  With the workaround I no 
> longer see the stall issue. I am not sure whether it is optimal solution 
> though. Let me know.

What does it look like?

The issue is very similar: 
https://mta.openssl.org/pipermail/openssl-dev/2016-March/005909.html


> 
> Thanks,
> 
> Grant
R,
Emeric

Re: Automatic Certificate Switching Idea

[Daniel Schneller]

> 
> That's perfect! Your feedback and possible trouble in doing this will
> also definitely help!
> 

Oh, if experience tells me one thing, no matter how “straightforward” this may 
look, there _will_ be trouble ;-)

Cheers
Daniel


-- 
Daniel Schneller
Principal Cloud Engineer
 
CenterDevice GmbH  | Hochstraße 11
   | 42697 Solingen
tel: +49 1754155711| Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431

Re: haproxy + RDP

[Antonio Trujillo Carmona]

El 12/05/17 a las 11:55, Aleksandar Lazic escribió:
> Hi Antonio Trujillo Carmona.
>
> Antonio Trujillo Carmona have written on Fri, 12 May 2017 10:23:59
> +0200:
>
>> El 11/05/17 a las 15:06, Aleksandar Lazic escribió:
>>> .../
>>> How about to activate the 'option tcp-check' as mentioned in the
>>> Warning?
>>> In the config below is it's commented, any reason why?
>>>
>>> It's also active in the doc which you maybe know.
>>>
>>> https://www.haproxy.com/doc/aloha/7.0/deployment_guides/microsoft_remote_desktop_services.html
>>>
>>> Does this changes anything?  
>> ok cleaing up a liter I try:
>> frontend RDP
>> mode tcp
>> bind *:3389
>> timeout client 1h
>> tcp-request inspect-delay 5s
>> tcp-request content accept if RDP_COOKIE
>> default_backend bk_rdp
>> #
>> backend bk_rdp
>> mode tcp
>> balance leastconn
>> #balance rdp_coockie
>> timeout server 1h
>> timeout connect 4s
>> log global
>> option tcplog
>> stick-table type string len 32 size 10k expire 1h peers pares
>> stick on rdp_cookie(msthash)
>> #   persist rdp-cookie
>> option tcp-check
>> #   option ssl-hello-chk
>> #   option tcpka
>> tcp-check connect port 3389 ssl
>>
>> #   server gr43sterminal01  10.104.22.142:3389 weight 1 check
>> verify none inter 2000 rise 2 fall 3
>> #   server gr43sterminal02  10.104.23.141:3389 weight 1 check
>> verify none inter 2000 rise 2 fall 3
>> #
>> default-server inter 3s rise 2 fall 3
>> server gr43sterminal01  10.104.22.142:3389 weight 1 check
>> server gr43sterminal02  10.104.23.141:3389 weight 1 check
>>
>> And I got:
>> [ALERT] 131/100222 (8564) : Proxy 'bk_rdp', server 'gr43sterminal01'
>> [/etc/haproxy/haproxy.cfg:189] verify is enabled by default but no CA
>> file specified. If you're running on a LAN where you're certain to
>> trust the server's certificate, please set an explicit 'verify none'
>> statement on the 'server' line, or use 'ssl-server-verify none' in
>> the global section to disable server-side verifications by default.
>> [ALERT] 131/100222 (8564) : Proxy 'bk_rdp', server 'gr43sterminal02'
>> [/etc/haproxy/haproxy.cfg:190] verify is enabled by default but no CA
>> file specified. If you're running on a LAN where you're certain to
>> trust the server's certificate, please set an explicit 'verify none'
>> statement on the 'server' line, or use 'ssl-server-verify none' in
>> the global section to disable server-side verifications by default.
>> [ALERT] 131/100222 (8564) : Fatal errors found in configuration.
>>
>> So I try adding verify none in server line
>>
>> and haproxy see both server up (but one is down).
>> I try  withou ssl:
>>
>> tcp-check connect port 3389
>> server gr43sterminal01  10.104.22.142:3389 weight 1 check
>> server gr43sterminal02  10.104.23.141:3389 weight 1 check
>>
>> but the result is the same haproxy see both server up (but one is
>> down)
>>
>> only if I leve only option tcp-check (or none) it seem work
>>
>>
>> #
>> #   persist rdp-cookie
>> option tcp-check
>> #   option ssl-hello-chk
>> #   option tcpka
>> #   tcp-check connect port 3389 ssl
>> #   tcp-check connect port 3389
>>
>> #   server gr43sterminal01  10.104.22.142:3389 weight 1 check
>> verify none inter 2000 rise 2 fall 3
>> #   server gr43sterminal02  10.104.23.141:3389 weight 1 check
>> verify none inter 2000 rise 2 fall 3
>> #
>> default-server inter 3s rise 2 fall 3
>> server gr43sterminal01  10.104.22.142:3389 weight 1 check
>> server gr43sterminal02  10.104.23.141:3389 weight 1 check
>> ##
>>
>>
>> output:
>>
>> [WARNING] 131/102105 (8773) : Server bk_rdp/gr43sterminal01 is DOWN,
>> reason: Layer4 timeout, info: " at initial connection step of
>> tcp-check", check duration: 3001ms. 1 active and 0 backup servers
>> left. 0 sessions active, 0 requeued, 0 remaining in queue.
> So finally it works.
>
> Regards
> Aleks
But in that mode it check server available, not service, if RDP service
is down haproxy don't detect it.


-- 

*Antonio Trujillo Carmona*

*Técnico de redes y sistemas.*

*Subdirección de Tecnologías de la Información y Comunicaciones*

Servicio Andaluz de Salud. Consejería de Salud de la Junta de Andalucía

_antonio.trujillo.sspa@juntadeandalucia.es_

Tel. +34 670947670 747670)