Re: OCSP renewal with 2.8

[Matthias Fechner]

Am 05.06.2023 um 10:08 schrieb William Lallemand:

As I explained in my previous mail, the option was not set on the bind
lines because of architectural problems, but you could expect to have a
way to do it globally in future versions.


thanks a lot for this information.
I will wait then to have a more simple, less verbose configuration 
(ideally just a single global parameter to enable ocsp renewal on bind line)
The scripts I use to renew OCSP are working for years without any 
problems, no need to rush here ;)


Gruß
Matthias

--

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: OCSP renewal with 2.8

[Matthias Fechner]

Am 02.06.2023 um 04:13 schrieb Shawn Heisey:
@Matthias I have no idea whether crt-list can load all certs in a 
directory like crt can.  If it can't, then you will probably need a 
script for starting/restarting haproxy that generates the cert list 
file.  If you wantthat script to be automatically run whenever someone 
does `systemctl restart haproxy`, you could use the ExecStartPre and 
ExecReloadPre options in a systemd service file to run your script.


My certificate files contain the server cert, the issuer cert, the 
private key, and DH PARAMETERS that are unique to that cert. 


maybe adding a global configuration parameter to enable ocsp retrieval 
for all certificates?


Adding an additional script does not make sense to me. If this would be 
required a keep with the current setup that proved to work fine.


Gruß
Matthias

--

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

OCSP renewal with 2.8

[Matthias Fechner]

Dear all,

I just saw in the release notes for 2.8 that an automatic OCSP renewal 
is now included and I would like to get rid of my manual scripts that 
are currently injecting the OCSP information.


I checked a little bit the documentation here:
https://docs.haproxy.org/2.8/configuration.html#ocsp-update
https://docs.haproxy.org/2.8/configuration.html#5.1-crt-list

and if I understood it correctly it only works if used together with a 
crt-list line.

I currently use the crt definition on a bind line like:
    frontend www-https
    mode tcp
    option tcplog
    bind 0.0.0.0:443 ssl crt /usr/local/etc/haproxy/certs/ 
alpn h2,http/1.1
    bind :::443 ssl crt /usr/local/etc/haproxy/certs/ alpn 
h2,http/1.1


Could you please help me, how I need to configure haproxy to use ocsp 
renewal.
It is not my intent to list all certificates in the haproxy 
configuration as that would make it unnecessarily complicated.


Thanks a lot.

Gruß
Matthias

--

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: [ANNOUNCE] haproxy-2.4.5

[Matthias Fechner]

Am 03.10.2021 um 08:53 schrieb Christopher Faulet:



Damned ! You're right...

It is a typo in the commit feca2a453 ("BUG/MINOR: filters: Always set 
FLT_END analyzer when CF_FLT_ANALYZE flag is set"). It also affects 
the 2.5-DEV. 


thanks a lot Christopher for the quick fix.
Just prepared a new package for FreeBSD and it is working fine.

I will ask the maintainer to include this patch till a new release is out.

Gruß
Matthias

--

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: [ANNOUNCE] haproxy-2.4.5

[Matthias Fechner]

Am 01.10.2021 um 18:09 schrieb Christopher Faulet:

HAProxy 2.4.5 was released on 2021/10/01. It added 69 new commits
after version 2.4.4. 



could it be, that this upgrade broke something.
The connection seems to hang.
I use the following configuration, which does not work anymore, 
downgrading haproxy to 2.4.4 fixes it:

global
    maxconn 2048
    hard-stop-after 5s
    #user haproxy
    #group haproxy
    user root
    daemon
    tune.ssl.default-dh-param 2048

    # logging
    ulimit-n 65536
    log /var/run/log local0 err

    # enable socket for ocsp reloading
    stats socket /var/run/haproxy.sock mode 600 level admin
    stats socket /var/run/haproxy_read.sock mode 666 level user
    stats timeout 2m

    # Configure chipers to not use, see 
https://mozilla.github.io/server-side-tls/ssl-config-generator/
    ssl-default-bind-ciphers 
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites 
TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options prefer-client-ciphers no-sslv3 
no-tlsv10 no-tlsv11 no-tls-tickets


    ssl-default-server-ciphers 
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-server-ciphersuites 
TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 
no-tls-tickets


    defaults
    mode http
    log global
    option httplog
    timeout connect 80s
    timeout client 80s
    timeout server 80s
    #timeout check 1s
    option forwardfor
    option http-server-close
    default-server inter 3s fall 2 rise 2 slowstart 60s
    compression algo gzip
    compression type text/html text/plain text/css

    frontend www-http
    bind *:80
    http-request add-header X-Forwarded-Proto http
    default_backend nginx-backend

    frontend www-https
    mode tcp
    option tcplog
    bind 0.0.0.0:443 ssl crt /usr/local/etc/haproxy/certs/ 
alpn h2,http/1.1
    bind :::443 ssl crt /usr/local/etc/haproxy/certs/ alpn 
h2,http/1.1


    tcp-request inspect-delay 5s
    tcp-request content accept if HTTP

    acl client_attempts_ssh payload(0,7) -m bin 5353482d322e30

    use_backend ssh if client_attempts_ssh

    use_backend nginx-http2-backend if { ssl_fc_alpn -i h2 }

    default_backend nginx-http-backend

    backend nginx-backend
    server www-1 127.0.0.1:8082 check send-proxy

    backend nginx-http2-backend
    mode tcp
    http-request add-header X-Forwarded-Proto https
    server www-1 127.0.0.1:8083 check send-proxy

    backend nginx-http-backend
    mode tcp
    server www-1 127.0.0.1:8082 check send-proxy

    backend ssh
    mode tcp
    option tcplog
    log global
    source 0.0.0.0 usesrc clientip
    server ssh 192.168.200.6:22
    timeout server 8h

    frontend haproxy-stats
    bind 192.168.0.251:9001
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /haproxy_stats
    stats auth _removed_:_removed_


Gruß
Matthias

--

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Using haproxy to have SSH in a HTTPS connection with HTX

[Matthias Fechner]

Dear all,

as HTTP2 is getting stable in haproxy 1.9.6 I decided to give it a try.
Currently I have the following setup:
    frontend www-https
    mode tcp
    option tcplog
    bind 0.0.0.0:443 ssl crt /usr/local/etc/haproxy/certs/
alpn h2,http/1.1
    bind :::443 ssl crt /usr/local/etc/haproxy/certs/ alpn
h2,http/1.1

    tcp-request inspect-delay 5s
    tcp-request content accept if HTTP

    acl client_attempts_ssh payload(0,7) -m bin 5353482d322e30
    use_backend ssh if client_attempts_ssh

    use_backend nginx-http2-backend if { ssl_fc_alpn -i h2 }
    default_backend nginx-http-backend

    backend nginx-http-backend
    mode tcp
    server www-1 127.0.0.1:8082 check send-proxy

    backend nginx-http2-backend
    mode tcp
    http-request add-header X-Forwarded-Proto https
    server www-1 127.0.0.1:8083 check send-proxy

    backend ssh
    mode tcp
    option tcplog
    source 0.0.0.0 usesrc clientip
    server ssh 192.168.200.6:22
    timeout server 8h

What I understood correctly from the documentation:
https://www.haproxy.com/de/blog/haproxy-1-9-has-arrived/

I must have the mode on http instead of tcp.

Is it possible to keep this ssh switch in place and use HTX for http
traffic?
(currently switching to http is not possible, as the mode for backend
and frontend must by equal, so I have to use tcp or http for both of them)
But if I switch to http, I cannot use the ssh backend anymore.

What do you recommend to get this solved (using another frontend you
forward the traffic to it?).

Thanks.

Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Haproxy 1.7.9 or earlier stopped using h2

[Matthias Fechner]

Dear all,

I configured some time ago haproxy to support http2 using tcp mode on
FreeBSD.

The configuration looked like:

frontend www-https
    mode tcp
    option tcplog
    bind : ssl crt /usr/local/etc/haproxy/certs/ alpn h2,http/1.1
    use_backend nginx-http2-backend if { ssl_fc_alpn -i h2 }
    default_backend nginx-http-backend

backend nginx-http2-backend
    mode tcp
    server www-1 127.0.0.1:8083 check send-proxy


Nginx on port 8083 is listing with this line:
listen 127.0.0.1:8083 http2 proxy_protocol;


That worked in the past. I have no idea when it stopped, but currently
it is not working anymore.
Is anyone aware of a change that could cause h2 to not working anymore?

Thanks

Matthias

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Hi Willy, Hi Dmitry,

Am 19.03.2017 um 12:40 schrieb Willy Tarreau:
> And here come two patches as a replacement for this temporary one. They
> are safer and have been done after throrough code review. I spotted a
> small tens of dirty corner cases having accumulated over the years due
> to the unclear meaning of the CO_FL_CONNECTED flag. They'll have to be
> addressed, but the current patches protect against these corner cases.
> They survived all tests involving delayed connections and checks with
> and without all handshake combinations, with tcp (immediate and delayed
> requests and responses) and http (immediate, delayed requests and responses
> and pipelining).
>
> I'm resending the first one you already got Dmitry to make things easier
> to follow for everyone. These three are to be applied on top of 1.7.3. I
> still have a few other issues to deal with regarding 1.7 before doing a
> new release (hopefully by the beginning of this week).
>

thank you both.
I have the patch running in PRD and face no problems so far.
I will deeply test them the next days and in case I find something I
will report it immediatly.


Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Dear Willy and Dmitry,

Am 14.03.17 um 22:17 schrieb Willy Tarreau:

Or you may prefer to wait for 1.7.4. It's not planned yet given that
there are other fixes in the wild waiting for some feedback though.

Thanks guys for the detailed feedback, it's now time to turn the page
and switch to less difficult ones!


I rolled now the 1.7.3 (version with the new patch) to the test 
environment and it seems to work reliably.


But the other message you wrote seems to point to another problem.

If you need any help for testing, please let me know.

Gruß,
Matthias

--
"Programming today is a race between software engineers striving to 
build bigger and better idiot-proof programs, and the universe trying to 
produce bigger and better idiots. So far, the universe is winning." -- 
Rich Cook

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Hi Willy,

Am 12.03.2017 um 14:51 schrieb Willy Tarreau:
> You're welcome. It's possible that I'll ask you to test a patch or two
> if I find anything suspicious, given that for now you're the first one
> to observe this issue.

sure, but next week I will not be able to test anything, so take your time.

I do not really understand why no one else has faced the problem.
Maybe no one has update yet (the new version went one week ago into
FreeBSD ports and here only to HEAD and not the quarterly branch).



Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Hi Willy, Hi Dmitry,

Am 12.03.2017 um 09:48 schrieb Willy Tarreau:
> Yep, that totally makes sense. Thanks for checking. Please find in
> attachment one which does properly apply here with -Rp1 (at least it
> will allow you to fix your production for the time it takes to find
> the root cause of this issue). If it doesn't apply it may mean that
> you have some leftovers of a previous patch or that your tree has
> other local patches, which could also explain the problem.

maybe the patch command from FreeBSD is different, I do not get your
patch applying correctly.
I always start from a clean build dir version 1.7.3.

But Dmitry already created an updated version of the port with the patch
included:
https://svnweb.freebsd.org/ports/head/net/haproxy/files/patch-src-proto_tcp.c?view=markup=435979

This patch applies correctly and I have now a test env running with the
patched version.
My monitoring system is already testing it and it looks promising, no
timeouts so far.
I also did some other tests that caused timeout before and everything
looks fine.

I will monitor it closely the next day and will report immediately if I
get a timeout, but I think it is fixed now.

Thanks you all for your support!


Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Hi Willy,

Am 11.03.2017 um 14:13 schrieb Willy Tarreau:
>
> OK so this is the one that I initially suspected and that after you
> reverted, didn't fix the issue for you.
>
> Are you sure you didn't have a problem when you reverted it ? (eg:
> failed to restart the process or something like this).
>
> At least now we're certain it's this one so we can try to imagine
> any possible corner case or any remaining bug this one could have
> uncovered.

I tested it again with the patch and I get the result:
patch -Rp1

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Am 11.03.2017 um 13:45 schrieb Willy Tarreau:
> I don't understand, the bisection didn't end ?
>
> Otherwise I'm inclined to think that the regression comes from
> "BUG/MEDIUM: tcp: don't poll for write when connect() succeeds", which
> it the one I proposed you to revert and which didn't change anything.
>
> Did "git bisect" end up telling you "This is the first bad commit" ?

I had a typo, here the result (typed git bisect goog instead of good):

cd4c5a3ecf5e77fb4734c423c914f7280199c763 is the first bad commit
commit cd4c5a3ecf5e77fb4734c423c914f7280199c763
Author: Willy Tarreau 
Date:   Wed Jan 25 14:12:22 2017 +0100

BUG/MEDIUM: tcp: don't poll for write when connect() succeeds

While testing a tcp_fastopen related change, it appeared that in the
rare
case where connect() can immediately succeed, we still subscribe to
write
notifications on the socket, causing the conn_fd_handler() to
immediately
be called and a second call to connect() to be attempted to double-check
the connection.

In fact this issue had already been met with unix sockets (which often
respond immediately) and partially addressed but incorrect so another
patch will follow. But for TCP nothing was done.

The fix consists in removing the WAIT_L4_CONN flag if connect() succeeds
and to subscribe for writes only if some handshakes or L4_CONN are still
needed. In addition in order not to fail raw TCP health checks, we have
to continue to enable polling for data when nothing is scheduled for
leaving and the connection is already established, otherwise the caller
will never be notified.

This fix should be backported to 1.7 and 1.6.
(cherry picked from commit 819efbf4b532d718abeb5e5aa6b2521ed725fe17)

:04 04 438a30304ac086b5cf0b0670253ff12efefeac82
6f2d45215d45de5669d3789eac247df856117e69 M  src

Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Hi Willy,

Am 07.03.2017 um 17:26 schrieb Willy Tarreau:
>
> So they're pretty much identical except the version. Are you interested in
> trying to do a bisection between 1.7.2 and 1.7.3 to find the culprit commit ?
> There are only 20 patches so it should take about 5 attempts so depending
> on the time it takes for the problem to appear it may be faster than
> speculating on each individual patch. If you're interested, the procedure
> is the following :
>
>   - from a git tree containing haroxy 1.7, you start bisection between 1.7.2
> and 1.7.3 this way :
>
> $ git bisect start v1.7.3 v1.7.2

here is the result:
idefix@server /u/p/n/h/w/haproxy-1.7.3> git bisect log
# bad: [9cb532a34ae190b350cdeb8bbbae25d524b10949] [RELEASE] Released
version 1.7.3
# good: [ddb646ee9182df570017ddf280873a1360a28898] [RELEASE] Released
version 1.7.2
git bisect start 'v1.7.3' 'v1.7.2'
# bad: [eaf96d7a0849b2883e98459f52489d555b6b013c] BUG/MAJOR: dns:
restart sockets after fork()
git bisect bad eaf96d7a0849b2883e98459f52489d555b6b013c
# bad: [cd4c5a3ecf5e77fb4734c423c914f7280199c763] BUG/MEDIUM: tcp: don't
poll for write when connect() succeeds
git bisect bad cd4c5a3ecf5e77fb4734c423c914f7280199c763
# good: [9a84d5dd772f3198da499534ac2f8d7694be2b23] BUILD: ssl: fix build
on OpenSSL 1.0.0
git bisect good 9a84d5dd772f3198da499534ac2f8d7694be2b23

Hope that helps you.

> I'm still wondering why you're the only one facing this for now and I
> suspect it's unrelated to the fact that you're on FreeBSD 11.
>

I face similar problems on FreeBSD 10.3-p11 but it is much harder to
find them.
On FreeBSD 10.3 if I try to access the main page:
https://domain/
it works without problem, but if I try to access:
https://domain/node/130/edit?destination=admin/content
It takes a very long time to get a response (but I get one).
I rolled back to haproxy 1.7.2 here too, and it solves the problem.

Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Am 2017-03-07 10:15, schrieb Willy Tarreau:

Thanks. In the mean time, I'm interested in trying to figure the code
paths we follow. Could please tell me :
  - if you're using send-proxy on your server lines ?


yes, all 2 backends have (but see first post, for full config:
backend nginx-http2-backend
mode tcp
server www-1 127.0.0.1:8083 check send-proxy

backend nginx-http-backend
mode tcp
server www-1 127.0.0.1:8082 check send-proxy


  - if you're using ssl on your server lines ?


ssl is terminated in the frontend, no ssl connection to the backend


  - if you're using SSL health checks ?


could you please see my first post, I'm not sure what you mean by this 
question, but I would says no, as the backend is not using ssl.



  - if you're only using HTTP or only using another TCP protocol, or
a combination of both ?


please see here also my first post for more details.
haproxy is working completely in tcp mode.
nginx is listening on port 8082 with http1 mode
nginx is listening on port 8083 with http2 mode
haproxy forwards the connection with acl:
acl http2 ssl_fc_alpn -i h2
use_backend nginx-http2-backend if http2
default_backend nginx-http-backend


--
Gruß
Matthias

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Am 2017-03-07 10:19, schrieb Willy Tarreau:

If I do a netstat -an I see a lot of:
tcp4   0  0 127.0.0.1.443  127.0.0.1.47010
TIME_WAIT
tcp4   0  0 127.0.0.1.443  127.0.0.1.46961
CLOSE_WAIT
tcp4   0  0 127.0.0.1.46961127.0.0.1.443  
FIN_WAIT_2


Is :443 your haproxy or the server after haproxy ? If it's the server 
behind
haproxy it means this one didn't respond to a close. If it's haproxy it 
means

it's stuck doing something else (very bad).


could you please check my first post, there is the complete haproxy 
config included, here a short summary:
sslh is listening on 192.168.0.251:443 and 192.168.1.2:443 and 
192.168.200.6:443 is acting as transparent proxy forwarding all requests 
to 192.168.0.251:8443 or 192.168.200.6:8443
External connections reaching the server on 
192.168.0.251/192.168.1.2/192.168.200.6 depending on which network 
interface they came in.
haproxy (frontend) is listening on localhost:443 and 192.168.0.251:8443, 
192.168.200.6:8443 (in tcp mode, as I use http2 to connect to backend 
nginx).


So in this case I would say that haproxy is hanging.
As this is a test system, I can do more tests if required.

Yes, please test with "nokqueue" in the global section, or start 
haproxy with
"-dk". It will switch to poll() and will tell us if there's a bug in 
the kqueue

poller. Please be aware that your CPU usage will increase a bit.


I tested it again.
haproxy 1.7.3 with patch reverted -> timeout
haproxy 1.7.3 with patch reverted and kqueue disabled -> timeout
ps shows: /usr/local/sbin/haproxy -dk -q -f /usr/local/etc/haproxy.conf 
-p /var/run/haproxy.pid

haproxy 1.7.2 -> no timeouts

I cannot says that in sslh is maybe a bug that is now triggered by a 
change inside haproxy.

The homepage of sslh is here: http://www.rutschle.net/tech/sslh.shtml

I use sslh to get ssh/https/openvpn to listen the same port (443). sslh 
just looks at the first bytes of the connection and acts as a 
transparent proxy to forward the connection to haproxy (https), ssh or 
openvpn.


Version 1.7.2 shows:
HA-Proxy version 1.7.2 2017/01/13
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = freebsd
  CPU = generic
  CC  = cc
  CFLAGS  = -O2 -pipe -fstack-protector -fno-strict-aliasing 
-DFREEBSD_PORTS
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 
USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1


Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 
200


Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")

Built with OpenSSL version : OpenSSL 1.0.2j  26 Sep 2016
Running on OpenSSL version : OpenSSL 1.0.2k  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : yes
Built without Lua support
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY

Available polling systems :
 kqueue : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available filters :
[SPOE] spoe
[TRACE] trace
[COMP] compression

Version 1.7.3 shows:
HA-Proxy version 1.7.3 2017/02/28
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = freebsd
  CPU = generic
  CC  = cc
  CFLAGS  = -O2 -pipe -fstack-protector -fno-strict-aliasing 
-DFREEBSD_PORTS
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 
USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1


Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 
200


Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")

Built with OpenSSL version : OpenSSL 1.0.2k  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.40 2017-01-11
Running on PCRE version : 8.40 2017-01-11
PCRE library supports JIT : yes
Built without Lua support
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY

Available polling systems :
 kqueue : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available filters :
[SPOE] spoe
[TRACE] trace
[COMP] compression


--
Gruß
Matthias

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Am 2017-03-07 00:32, schrieb Willy Tarreau:
  patch -Rp1 < 
0001-BUG-MEDIUM-tcp-don-t-poll-for-write-when-connect-suc.patch


I've just tested here on 1.7.3 and it does apply correctly.

With git apply you'll have to pass -R as well.
Sorry for not being clear the first time.


so, shortly after I started to 1.7.3 with the patch reverted I got 
timeouts again.

If I do a netstat -an I see a lot of:
tcp4   0  0 127.0.0.1.443  127.0.0.1.47010
TIME_WAIT
tcp4   0  0 127.0.0.1.443  127.0.0.1.46961
CLOSE_WAIT
tcp4   0  0 127.0.0.1.46961127.0.0.1.443  
FIN_WAIT_2


Can I do anything to help you to track the problem to the source?

--
Gruß
Matthias

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Thanks Willy,

Am 07.03.2017 um 00:32 schrieb Willy Tarreau:
> Sorry, when I said "revert" I meant typically like this :
>
>   patch -Rp1 < 0001-BUG-MEDIUM-tcp-don-t-poll-for-write-when-connect-suc.patch
>
> I've just tested here on 1.7.3 and it does apply correctly.
>
> With git apply you'll have to pass -R as well.
> Sorry for not being clear the first time.

I recompiled haproxy 1.7.3 with the patch reverted and will test it now.

Please give me 9 hours, to give you a real feedback, but so far the
monitoring system is quiet.

Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Dear Willy and Dmitry,

Am 06.03.2017 um 11:16 schrieb Willy Tarreau:
> with the attachment now (thanks Dmitry)

hm, I'm not able to apply the patch:
git apply --ignore-space-change --ignore-whitespace
0001-BUG-MEDIUM-tcp-don-t-poll-for-write-when-connect-suc.patch

But I get:
0001-BUG-MEDIUM-tcp-don-t-poll-for-write-when-connect-suc.patch:41:
trailing whitespace.
if (connect(fd, (struct sockaddr *)>addr.to,
get_addr_len(>addr.to)) == -1) {
0001-BUG-MEDIUM-tcp-don-t-poll-for-write-when-connect-suc.patch:42:
trailing whitespace.
if (errno == EINPROGRESS || errno == EALREADY) {
0001-BUG-MEDIUM-tcp-don-t-poll-for-write-when-connect-suc.patch:43:
trailing whitespace.
/* common case, let's wait for connect status */
0001-BUG-MEDIUM-tcp-don-t-poll-for-write-when-connect-suc.patch:44:
trailing whitespace.
conn->flags |= CO_FL_WAIT_L4_CONN;
0001-BUG-MEDIUM-tcp-don-t-poll-for-write-when-connect-suc.patch:45:
trailing whitespace.
}
error: patch failed: src/proto_tcp.c:474
error: src/proto_tcp.c: patch does not apply

It is a 1.7.3 version (I just did a make extract in the freebsd port and
tried to apply the patch).

Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Dear Rainer,

Am 06.03.2017 um 09:52 schrieb rai...@ultra-secure.de:
> it would be cool if somebody could open a PR at
>
> https://bugs.freebsd.org/
>
> I personally don't use FreeBSD 11 for any of my HAProxy-installations
> (yet), so I'm not really affected (yet) - but thanks for the heads-up. 

I opened a bug report here:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217576


I have only one server already upgraded to FreeBSD 11. The 10.3
installation are running fine with haproxy 1.7.3.

Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Hi Georg,

Am 06.03.2017 um 09:43 schrieb Georg Faerber:
> I'm not running FreeBSD myself, but have a look at [1]: In the
> follow-ups to this thread there are two more people reporting problems.
>
> [1] https://www.mail-archive.com/haproxy@formilux.org/msg25093.html

no, this cannot be the problem, because this error reported in [1] is
related to haproxy version 1.7.2.

My problem is related to 1.7.3. The problem was introduced by a change
for 1.7.3. as 1.7.2 is running fine.


Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook




signature.asc
Description: OpenPGP digital signature

Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Matthias Fechner]

Dear all,


are problem with haproxy 1.7.3 on FreeBSD 11.0-p8 known?

I have the problem that I got a lot of timeout for all websites that are
behind haproxy.

Haproxy does terminate the SSL connection and forwards to nginx. Before
haproxy I have a sslh running.


Downgrading to version 1.7.2 fixed the problem.


Here my config (I removed some fqdns and username and password):
global
maxconn 2048
user haproxy
group haproxy
daemon
tune.ssl.default-dh-param 2048

# logging
ulimit-n 65536
#log /var/run/log local0 info
log /var/run/log local0 err

# Configure chipers to not use, see
https://mozilla.github.io/server-side-tls/ssl-config-generator/
ssl-default-bind-ciphers
ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
ssl-default-bind-options no-sslv3 no-tls-tickets
ssl-default-server-ciphers
ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
ssl-default-server-options no-sslv3 no-tls-tickets

defaults
mode http
log global
option httplog
#option dontlog-normal
timeout connect 80s
timeout client 80s
timeout server 80s
#timeout check 1s
#timeout http-keep-alive 1s
#timeout http-request 400s # slowloris protection
option forwardfor
option http-server-close
default-server inter 3s fall 2 rise 2 slowstart 60s
compression algo gzip
compression type text/html text/plain text/css

frontend www-http
bind *:80
redirect scheme https code 301 if !{ ssl_fc }
reqadd X-Forwarded-Proto:\ http
default_backend nginx-backend

frontend www-https
mode tcp
bind 192.168.0.251:8443 ssl crt
/usr/local/etc/haproxy/certs/ alpn h2,http/1.1
bind 192.168.200.6:8443 ssl crt
/usr/local/etc/haproxy/certs/ alpn h2,http/1.1
bind localhost:443 ssl crt /usr/local/etc/haproxy/certs/
alpn h2,http/1.1
bind 127.0.0.1:443 ssl crt /usr/local/etc/haproxy/certs/
alpn h2,http/1.1

acl use_nginx ssl_fc_sni -i fqdn1 fqdn2
acl http2 ssl_fc_alpn -i h2
use_backend nginx-http2-backend if http2
use_backend nginx-http-backend if use_nginx

default_backend nginx-http-backend

backend nginx-backend
server www-1 127.0.0.1:8082 check send-proxy

backend nginx-http2-backend
mode tcp
server www-1 127.0.0.1:8083 check send-proxy

backend nginx-http-backend
mode tcp
server www-1 127.0.0.1:8082 check send-proxy

frontend haproxy-stats
bind 192.168.0.251:9001
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /haproxy_stats
stats auth :

Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: Inform backend about https for http2 connections

[Matthias Fechner]

Am 06.08.2016 um 15:12 schrieb Neil - HAProxy List:
>
> if you can have the app not specify the scheme for the css etc.  just use
>
> //site.com/path 
> or
> /path if it is on the same site
>

as I do not develop the apps I cannot do it.

The configuration how the return these paths are defined by the fastcgi
parameters passed.

I think the only possibilty whould then to define several backends on
different ports and define there the h2c or h2 in the frontend
configuration of nginx.

To check if for the TLS connection it is a h2 enabled client I can use:

acl http2 ssl_fc_alpn -i h2


Is there a similar check for the not TLS connection I can use?


Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: Inform backend about https for http2 connections

[Matthias Fechner]

Am 06.08.2016 um 05:31 schrieb Igor Cicimov:
> Afaik, since http2 is by default tls encrypted just by specifying h2
> as protocol to the backend should be enough i guess.

this is not completely true. There is h2c which uses http2 without TLS.


Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Inform backend about https for http2 connections

[Matthias Fechner]

Dear all,


I use haproxy in tcp mode to have http2 working.
Now I have the problem that the backend has to know if the connection
was encrypted or not (some websites using this information to add the
schema to css and javascript URIs).

Normally I think a
reqadd X-Forwarded-Proto:\ https

should do the trick.

Will this work if working in tcp mode or are there other tricks to do this?


Thanks
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: Haproxy with SNI and http2 seperation

[Matthias Fechner]

Hi Cyril,

Am 19.07.2016 um 00:27 schrieb Cyril Bonté:
> You probably have an issue here : hdr(host) won't work with mode tcp.
> If you wan't to check the domain provided by SNI, you should use
> req.ssl_sni instead.
>
> Have a look to the example provided in the documentation :
> http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#req.ssl_sni

thanks a lot!
I got it working using ssl_fc_sni.
As haproxy terminates the ssl connection I think it should be save to
use ssl_fc_sni.

Or is there I reason I should prefer req_ssl_sni instead?


Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Haproxy with SNI and http2 seperation

[Matthias Fechner]

Dear all,

I use haproxy 1.6.7 on a FreeBSD host.

I just try to do the following:
Check if hostname is domain1 or domain2 or bladomain3 or ... , mark it
to send the request to nginx
 all other requests should go to apache backend.

Check if client support http2, if yes and nginx acl matches, send it to
nginx with a special port.

All the rest should go to apache backend.

I tried to do this using two acls and 3 backends (apache, nginx,
nginx-http2).

But I could not get it working and I'm not sure what I'm doing wrong.
Here my frontend and backend definition:

frontend www-https
mode tcp

bind 192.168.0.251:8443 ssl crt /usr/local/etc/haproxy/certs/
alpn h2,http/1.1
bind 192.168.200.6:8443 ssl crt /usr/local/etc/haproxy/certs/
alpn h2,http/1.1
bind localhost:443 ssl crt /usr/local/etc/haproxy/certs/ alpn
h2,http/1.1
bind 127.0.0.1:443 ssl crt /usr/local/etc/haproxy/certs/ alpn
h2,http/1.1

acl use_nginx hdr(host) -i domain1 domain2 bladomain3
acl http2 ssl_fc_alpn -i h2

use_backend nginx-http2-backend if use_nginx http2
use_backend nginx-http-backend if use_nginx

default_backend apache-backend

backend apache-backend
server www-1 127.0.0.1:8081 check

backend nginx-http-backend
server www-1 127.0.0.1:8082 check


backend nginx-http2-backend
mode tcp
http-request add-header X-Forwarded-Proto https
server www-1 127.0.0.1:8083 check send-proxy


Can please someone point me to the mistake I did?
Thanks a lot!

Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook