postfix with postscreen behind haproxy
Hi, I am running (trying to, really) postfix with postscreen (really Zimbra) behind haproxy. I enabled proxy protocol support in Zimbra 08:27:26 (TEST) zimbra@zcs-fe5 [~] $ postconf |grep haproxy postscreen_upstream_proxy_protocol = haproxy 08:26:47 (TEST) zimbra@zcs-fe6 [~] $ postconf |grep haproxy postscreen_upstream_proxy_protocol = haproxy and have the following backend configuration in haproxy: backend smtp-zimbra-backend balance roundrobin mode tcp timeout server 1m timeout connect 10s option smtpchk EHLO lb.server.bla #option smtpchk server pm01 192.168.185.206:25 check send-proxy inter 45s fastinter 2s downinter 2s server pm02 192.168.185.207:25 check send-proxy inter 45s fastinter 2s downinter 2s However, this yields a „Layer 7 invalid response: „200-lb.server.bla“ From this article, I deduce that one cannot use smtpcheck with postscreen? https://www.linuxbabe.com/mail-server/smtp-imap-proxy-with-haproxy-debian-ubuntu-centos Set Up SMTP and IMAP Proxy with HAProxy (Debian, Ubuntu, CentOS) linuxbabe.com Does anybody have some experience with this? I would really prefer to have „smtpcheck“ to have some sort of health check to remove non-working backends from the pool… Rainer
Re: Transparent proxy issue on FreeBSD
> Am 07.03.2023 um 18:26 schrieb Marc West : > > On 2023-03-07 08:09:04, Rainer Duffner wrote: >> I admit I only toyed with TP, so I really don???t know what I???m doing >> there, but: >> >> Have you tried to just use pfSense for this? The developer of the package >> (https://github.com/PiBa-NL) seemed to be active here, but I haven???t seen >> anything from him since 2020, so I wonder if he has moved on. >> >> My co-workers use OPNSense for this purpose - and on VMWare, they insist >> that only em(4) NICs work. >> >> >> If you don???t find his email-address, I can mail it to you. > > Thanks for the suggestion. I haven't tried HAProxy on pfSense but the > working transparent config and related ipfw fwd rules we have did come > from PiBa-NL [1]. Ah, ok. Either ask on the freebsd-forum or the mailing-list - or try with OPNSense/pfSense and if the problem persists, you might get more response on the forums there. pf and ipfw are very specialized parts of the kernel and very few developers want to touch it, AFAIK. > Everything does function perfectly until a brief > period with production traffic and something happens to cause the tproxy > bind errors and request failures to start. I'm just not sure what is > going wrong or how to debug further. > > [1] https://www.mail-archive.com/haproxy@formilux.org/msg09923.html >
Re: Transparent proxy issue on FreeBSD
> Am 07.03.2023 um 08:46 schrieb Marc West : > > > > Any other thoughts to look at or data that would be helpful to collect? > I admit I only toyed with TP, so I really don’t know what I’m doing there, but: Have you tried to just use pfSense for this? The developer of the package (https://github.com/PiBa-NL) seemed to be active here, but I haven’t seen anything from him since 2020, so I wonder if he has moved on. My co-workers use OPNSense for this purpose - and on VMWare, they insist that only em(4) NICs work. If you don’t find his email-address, I can mail it to you.
haproxy and CARP - binding a frontend to a specific IP on the backup-server
Hi, I run two FreeBSD 12.2 servers with haproxy 2.0.22 in a CARP setup. The frontend-interfaces have multiple IPs and I need to have this statement in at least one backend service: source 192.168.185.29 This is because the target-service has some whitelisting for this specific address. This has worked well over several years, however recently (maybe with the upgrade to the 12-series - I can't be sure), it seems that haproxy on the CARP BACKUP cannot use this configuration anymore - which, upon closer look, is not totally unreasonable and technically correct. I assume, previously it would just ignore the statement and use the interface IP. I've now commented it out on the slave, but it's a bit silly to have to remember to "fix" the slave manually on switch-over. Is there a way to get haproxy to just ignore the source-statement if it can't use that address for sending (which it obviously can't, when it's not MASTER)? Rainer
Re: OT: About WebPageTest results (was Re: SSL Labs says my server isn't doing ssl session resumption)
> Am 21.06.2021 um 18:25 schrieb Shawn Heisey : > > On 2021-06-20 06:03, Shawn Heisey wrote: >> Unrelated, and off topic because it's mostly about Apache, but strange: >> I've been doing some tests with webpagetest.org, and seeing REALLY >> long load times for some resources in their waterfall graph. I see no >> speed problems when I load the pages from my workstation at home. > > Followup on this, information which others here might find useful: > > By default WebPageTest defaults to traffic shaping of 5 Mbps down and 1 Mbps > up, which it thinks simulates a cable connection. That's laughable -- I get > 460 Mbps down and 12 Mpbs up on my cable connection, and I'm not even paying > for the maximum bandwidth I COULD get. > > Long story short, hitting a web page with about 25 megabytes of images takes > over 40 seconds for WebPageTest to render. If I switch from that default > "5/1 Mbps Cable" traffic shaping to native (no traffic shaping at all) the > render takes 1.8 seconds, which is approximately what I see when I hit the > page myself. Server in AWS. > > When I do the math, 40 seconds is actually quite fast for downloading those > images on a 5 megabit connection. So there was no actual problem. WBT needs > to make the choice of traffic shaping a lot more prominent, and provide more > realistic options than what they have at the moment. To even see bandwidth > options, you have to open advanced settings. And the only option I could see > in their list that's faster than the default (aside from native) is FIOS, > which they've got at 20Mb down and 5Mb up. They have forums, I'll make > suggestions there. > > Thanks, > Shawn > It’s probably to make DDoSes more difficult (like basically everything these days) I never got around to host my own WPT instance (for work). I mainly use the public version to to get „a feeling“ for the speed and to weed out any caching effects of local browsers with pages too complex to use curl or httpie….
Re: Question about SNI
Am 2019-06-25 19:44, schrieb Lukas Tribus: Hello Rainer, [...] I suggest your try a HEAD request for the haproxy health check instead: option httpchk HEAD /swagger/ui/index HTTP/1.1\r\nHost:\ app-api.dom.intern\r\nUser-agent:\ LB-Check-API\r\nConnection:\ close There is no need for the actual HTTP payload to be send to haproxy, and I don't recall what happens when the response is bigger than haproxy buffers. And now it works! Holy moly. I've never had this problem. Always used GET. Though we also never used SNI with haproxy. (You know how you carry stuff over from the past? When I was first introduced to haproxy, my then co-worker used GET in the healthchecks. It kind-of stuck.) I know that once you add SSL to something, things can get very dicey and the smallest details become relevant. The problem in this case may also be that the result of the GET is a bit larger than usual (in most cases, we have dedicated health-check pages that just return "ok" and nothing else). Thanks a lot for your help. It was invaluable! Best Regards Rainer
Re: Question about SNI
Am 2019-06-25 18:26, schrieb Lukas Tribus: Hell Rainer, On Tue, 25 Jun 2019 at 18:01, wrote: Ah, OK. Thanks. However, I still get L7TOUT on the healthchecks. I don't follow. Are health checks working or not? You started this thread saying: Healthchecks are OK. But running a curl gives 503 So, are health checks working and does haproxy consider your backend servers up or not? Oh, sorry. I tried a lot of things over the last days Currently, with the last config I posted (and keepalived disabled), I get L7TOUTs from the backend. However, the healthchecks do arrive on the actual servers and the servers answer with a code 200. When the actual app is removed (and replaced by a static file), the healthchecks also work and the backend is OK for haproxy. So, as I said, I wonder what else haproxy expects. Best Regards Rainer
Re: Question about SNI
Am 2019-06-25 16:54, schrieb Lukas Tribus: Hello Rainer, On Tue, 25 Jun 2019 at 16:18, wrote: The requests from the healthchecks *do* arrive at the right vhosts on the backend, there's a code 200 in the logs. So, I wonder what exactly is timing out for haproxy. The server on the other end does not accept non-SNI connections, there's an SSL handshake error if you don't do SNI. At this point, I feel pretty dumb. Health checks are fine. Actual requests with your production traffic are not. check-sni specifies the SNI value to use for health-checks ONLY. sni specifies the SNI value for actual traffic ONLY. You need both though. So you servers look like this for example, notice that both check-sni and sni is configured: server server1 10.10.10.11:443 check check-ssl ssl verify none force-tlsv12 maxconn 3000 cookie s1 check-sni host3.intern sni str(host3.intern) server server2 10.10.10.12:443 check check-ssl ssl verify none force-tlsv12 maxconn 3000 cookie s2 check-sni host3.intern sni str(host3.intern) If you only configure check-sni, only health checks work, production traffic will not. Ah, OK. Thanks. However, I still get L7TOUT on the healthchecks. This is the haproxy that came with Ubuntu 18.04.2, haproxy 1.8.8. At this point, I wonder if the keepalive configuration on the host has any influence on this. Though, it could also be an IIS-thing. If my co-worker removes the application from the app-server, the healthcheck turns ok. time curl -kv -I --resolve "app-api.dom.intern:443:10.200.16.36" -H "Host: app-api.dom.intern" -o /dev/null https://app-api.dom.intern:443/swagger/ui/index * Added app-api.dom.intern:443:10.200.16.36 to DNS cache * Hostname app-api.dom.intern was found in DNS cache * Trying 10.200.16.36... * TCP_NODELAY set % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to app-api.dom.intern (10.200.16.36) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs } [5 bytes data] * TLSv1.2 (OUT), TLS handshake, Client hello (1): } [218 bytes data] * TLSv1.2 (IN), TLS handshake, Server hello (2): { [98 bytes data] * TLSv1.2 (IN), TLS handshake, Certificate (11): { [827 bytes data] * TLSv1.2 (IN), TLS handshake, Server key exchange (12): { [300 bytes data] * TLSv1.2 (IN), TLS handshake, Server finished (14): { [4 bytes data] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): } [37 bytes data] * TLSv1.2 (OUT), TLS change cipher, Client hello (1): } [1 bytes data] * TLSv1.2 (OUT), TLS handshake, Finished (20): } [16 bytes data] * TLSv1.2 (IN), TLS handshake, Finished (20): { [16 bytes data] * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.dom.intern * start date: Jun 19 11:49:25 2019 GMT * expire date: Jun 19 11:59:26 2039 GMT * issuer: CN=*.dom.intern * SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 } [5 bytes data] * Using Stream ID: 1 (easy handle 0x55ed4bbd0900) } [5 bytes data] HEAD /swagger/ui/index HTTP/2 Host: app-api.dom.intern User-Agent: curl/7.58.0 Accept: */* { [5 bytes data] * Connection state changed (MAX_CONCURRENT_STREAMS updated)! } [5 bytes data] < HTTP/2 200 < cache-control: no-cache < pragma: no-cache < content-type: text/html < expires: -1 < server: Microsoft-IIS/10.0 < x-aspnet-version: 4.0.30319 < x-content-type-options: nosniff < x-xss-protection: 1; mode=block < x-frame-options: SAMEORIGIN < access-control-allow-origin: https://app-bo.dom.intern < access-control-allow-headers: Origin, X-Requested-With, Content-Type, Authorization, X-Token-Jwt < access-control-allow-credentials: true < access-control-allow-methods: * < date: Tue, 25 Jun 2019 15:52:40 GMT < 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0 * Connection #0 to host app-api.dom.intern left intact real0m0.034s user0m0.011s sys 0m0.008s backend app_api mode http server ISOPROD036 10.200.16.36:443 check check-ssl ssl verify none force-tlsv12 maxconn 3000 cookie s1 check-sni app-api.dom.intern sni str(app-api.dom.intern) server ISOPROD037 10.200.16.37:443 check check-ssl ssl verify none force-tlsv12 maxconn 3000 cookie s2 check-sni app-api.dom.intern sni str(app-api.dom.intern) option httpclose option forwardfor option httpchk GET /swagger/ui/index HTTP/1.1\r\nHost:\ app-api.dom.intern\r\nUser-agent:\ LB-Check-API\
Re: Question about SNI
Am 2019-06-25 14:44, schrieb Lukas Tribus: Hello Rainer, On Tue, 25 Jun 2019 at 12:53, wrote: Hi, I tried to read up on this but there are many examples and not all of them seem "correct". It's simple: do not content-switch based on SNI. Use the host header instead. That's it. OK, I switched that out. But it's really only a 2ndary-problem at the moment, because I still get L7 timeouts on the backends. The example I provided for the backend is actually not what I wanted to write. I was under the assumption that all that was needed to enable sni for healthchecks on a (http) backend was to say "check-sni" servername.bla.server http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#check-sni backend app_api mode http server PROD036 10.200.16.36:443 check check-ssl ssl verify none force-tlsv12 maxconn 3000 cookie s1 check-sni api.app.intern server PROD037 10.200.16.37:443 check check-ssl ssl verify none force-tlsv12 maxconn 3000 cookie s2 check-sni api.app.intern option httpclose option forwardfor option httpchk GET / HTTP/1.1\r\nHost:\ api.app.intern\r\nUser-agent:\ LB-Check-API\r\nConnection:\ close http-check expect string Hello http-check disable-on-404 cookie SERVERID insert indirect nocache httponly balance leastconn #stick-table type string len 52 size 100k expire 60m The requests from the healthchecks *do* arrive at the right vhosts on the backend, there's a code 200 in the logs. So, I wonder what exactly is timing out for haproxy. The server on the other end does not accept non-SNI connections, there's an SSL handshake error if you don't do SNI. At this point, I feel pretty dumb. I really appreciate your help. Best Regards Rainer
Re: Question about SNI
Am 2019-06-20 15:38, schrieb Lukas Tribus: Hello, On Thu, 20 Jun 2019 at 14:49, wrote: I now used ssl_fc_sni_reg -i host3.intern I hope, this is also OK. It's not. You are already doing the right thing in the frontend, by content switching based on the host header and not based on the SNI, so please, don't rely on frontend SNI in your backend. Search the mailing list archives if you want to know why that's a bad idea. Lukas Hi, I tried to read up on this but there are many examples and not all of them seem "correct". I've got the following config now: frontend app_frontend mode http bind *:80 bind 10.200.16.10:443 ssl crt /etc/haproxy/ssl/star.theapp.intern.pem maxconn 2000 use_backend app_api if { ssl_fc_sni_reg -i app-api.theapp.intern } use_backend app_admin_servicesif { ssl_fc_sni_reg -i app-admin-services.theapp.intern } use_backend app_dms_services if { ssl_fc_sni_reg -i app-dms-services.theapp.intern } use_backend app_external_services if { ssl_fc_sni_reg -i app-external-services.theapp.intern } use_backend app_boif { ssl_fc_sni_reg -i app-bo.theapp.intern } use_backend app_scheduler if { ssl_fc_sni_reg -i app-scheduler.theapp.intern } #use_backend app_api if { sni hdr(host) -i app-api.theapp.intern } #use_backend app_admin_servicesif { sni hdr(host) -i app-admin-services.theapp.intern } #use_backend app_dms_services if { sni hdr(host) -i app-dms-services.theapp.intern } #use_backend app_external_services if { sni hdr(host) -i app-external-services.theapp.intern } #use_backend app_boif { sni hdr(host) -i app-bo.theapp.intern } #use_backend app_scheduler if { snd hdr(host) -i app-scheduler.theapp.intern } capture request header User-Agent len 500 redirect scheme https code 301 if !{ ssl_fc } http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Ssl on if { ssl_fc } backend app_api mode http server PROD036 10.200.16.36:443 check check-ssl ssl verify none force-tlsv12 maxconn 3000 cookie s1 sni hdr(app-api.theapp.intern) check-sni app-api.theapp.intern server PROD037 10.200.16.37:443 check check-ssl ssl verify none force-tlsv12 maxconn 3000 cookie s2 sni hdr(app-api.theapp.intern) check-sni app-api.theapp.intern option httpclose option forwardfor option httpchk GET / HTTP/1.1\r\nHost:\ app-api.theapp.intern\r\nConnection:\ close http-check expect string Hello http-check disable-on-404 cookie SERVERID insert indirect nocache httponly balance leastconn This gets me a L7 timeout on the backend-servers. curl-ing the URLs works without problems. Because it's all encrypted, I have a hard time figuring out what haproxy is actually sending to the backend. Is there a way to enable some sort of logging on what requests are actually made to the backend? Best Regards Rainer
Re: Question about SNI
Am 2019-06-20 13:18, schrieb Lukas Tribus: Hello, you only enabled SNI for health checks (check-sni). You need to enable SNI for the actual traffic with the sni keyword. sni str(intern3.local) or sni hdr(host) lukas Ah, ok. Thanks a lot! I now used ssl_fc_sni_reg -i host3.intern I hope, this is also OK.
Question about SNI
Hi, likely, I'm the one doing something wrong, but I can't figure it out. I have the following configuration: frontend the_frontend mode http bind *:80 bind *:443 ssl crt /etc/haproxy/ssl/star.intern.pem maxconn 2000 use_backend host1 if { hdr_dom(host) -i host1.intern } use_backend host2 if { hdr_dom(host) -i host2.intern } use_backend host3 if { hdr_dom(host) -i host3.intern } use_backend host4 if { hdr_dom(host) -i host4.intern } use_backend host5 if { hdr_dom(host) -i host5.intern } capture request header User-Agent len 500 redirect scheme https code 301 if !{ ssl_fc } http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Ssl on if { ssl_fc } http-response set-header Strict-Transport-Security max-age=15768000 http-response set-header X-Server %s backend host3 mode http server server1 10.10.10.11:443 check check-ssl ssl verify none force-tlsv12 maxconn 3000 cookie s1 check-sni host3.intern server server2 10.10.10.12:443 check check-ssl ssl verify none force-tlsv12 maxconn 3000 cookie s2 check-sni host3.intern option httpclose option forwardfor option httpchk GET / HTTP/1.1\r\nHost:\ host3.intern\r\nConnection:\ close http-check expect string Hello http-check disable-on-404 cookie SERVERID insert indirect nocache httponly balance leastconn stick-table type string len 52 size 100k expire 60m Healthchecks are OK. But running a curl gives 503 curl -kv --resolve "host3.intern:443:10.10.10.1" -H "Host: host3.intern" https://host3.intern:443/bla2 The other side are IIS-hosts and my co-worker says if he disabled SNI on one host, the rest starts working. Not sure if the stick-table line is actually correct (someone else built this). On of my configs, the stick-table config is a bit larger, like this: stick-table type string len 52 size 100k expire 60m stick store-response res.cook(JSESSIONID) stick on req.cook(JSESSIONID) But it should not be relevant to the error, right? Anyone got any ideas? Regards Rainer
Re: NFS mounts freezing via Haproxy
> Am 22.05.2018 um 06:46 schrieb TomK: > > Trying to mount an NFS share vi an Haproxy / Keepalived configuration. When I > mount the NFS share directly from the host, bypassing Haproxy / Keepalived, > it works fine. However, when I try via the Haproxy / Keepalived combination, > it freezes. Maybe I’m a little slow - but what exactly is this config trying to achieve?
Question about haproxy logs
Hi, I have lines like these: Apr 19 09:32:03 lb-prod haproxy[16717]: 127.0.0.1:50898 [19/Apr/2018:09:32:03.174] srv-pub-front-ssl srv-pub-back-ssl/WINSRV 0/0/0/36/290 500 284 - - --VN 3/1/0/1/0 0/0 "POST /SaveStatistics HTTP/1.1" Does that mean that the backend-server (WINSRV) replied with a code 500?
Re: What is a nice way to bypass the maintenance mode for certain IP's?
Am 2018-02-20 13:44, schrieb Willy Tarreau: On Tue, Feb 20, 2018 at 12:33:59PM +0100, rai...@ultra-secure.de wrote: can you point out what is wrong with this config? https://stackoverflow.com/questions/29248144/working-configuration-for-haproxy-with-the-force-persist-setting Thanks for the link, I've responded there so that the response can be found for future readers. Willy Thank you! Best Regards Rainer
Re: What is a nice way to bypass the maintenance mode for certain IP's?
Am 2018-02-19 14:04, schrieb Willy Tarreau: Hi, On Mon, Feb 19, 2018 at 12:18:36PM +, Pieter Vogelaar wrote: Hi, At the moment if we set backends in maintenance mode, the servers can't be reached by anyone. Is it possible to still allow traffic from certain IP's (of the office network) so that testing can be done, before the backend is available to the general public again? Please take a look at "force-persist", it's designed exactly for what you want to do. Regards, Willy Hi, can you point out what is wrong with this config? https://stackoverflow.com/questions/29248144/working-configuration-for-haproxy-with-the-force-persist-setting This pretty much how I would end up doing it and I'm curious to know if there are any errors in my thinking. (haproxy 1.7.9) Regards Rainer
Re: Problem with BOM in healthcheck-file?
Am 2017-07-20 14:18, schrieb Jarno Huuskonen: Can you share how you've configured health checks in haproxy.cfg ? backend site-back balance roundrobin mode http option httpchk GET /healthcheck.htm HTTP/1.1\r\nHost:\ site.com\r\nConnection:\ close http-check expect string server_up http-check disable-on-404 cookie SERVERID insert indirect nocache server SERVER1 ip1:80 check maxconn 3000 cookie s1 server SERVER2 ip2:80 check maxconn 3000 cookie s2 And can you show curl -v output of the BOM response and (lb2-beeline-prod ) 0 # curl -v -H "Host: site.com" http://ip1:80//Healthcheck_broken.htm * Trying ip1... * TCP_NODELAY set * Connected to ip1 (ip1) port 80 (#0) GET //Healthcheck_broken.htm HTTP/1.1 Host: site.com User-Agent: curl/7.50.3 Accept: */* < HTTP/1.1 200 OK < Content-Type: text/html < Last-Modified: Thu, 20 Jul 2017 15:18:51 GMT < Accept-Ranges: bytes < ETag: "4d832b7e6b1d31:0" < Server: Microsoft-IIS/8.5 < Date: Thu, 20 Jul 2017 15:27:08 GMT < Content-Length: 24 < ��server_up * Curl_http_done: called premature == 0 * Connection #0 to host ip1 left intact od -c bomfile. 000 377 376 s \0 e \0 r \0 v \0 e \0 r \0 _ \0 020 u \0 p \0 \r \0 \n \0 030
Problem with BOM in healthcheck-file?
Hi, I had a very strange situation earlier. He have a site behind a haproxy/nginx combination, it has been working for years. (It's Windows). However, suddenly I get L7 timeouts. But curl to the healthcheck URL works perfectly. The output I get looks like this: ��server_up Piping curl through cat -v looks like this: M-^?M-~s^@e^@r^@v^@e^@r^@_^@u^@p^@^M^@ ^@ They have started to generate this file via a powershell script and it has inserted a byte-order-mark into the file (BOM). Took a while to figure this out ;-) haproxy 1.6.9 (FreeBSD 10-amd64) seems to actually choke on these files and hang - I had to kill -9 it. After replacing the file with its previous, ASCII-only copy, everything started to work again. Can anyone reproduce this? Maybe it's fixed in later versions? Regards Rainer
Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8
Am 2017-03-06 10:05, schrieb Matthias Fechner: Dear Rainer, I opened a bug report here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217576 I have only one server already upgraded to FreeBSD 11. The 10.3 installation are running fine with haproxy 1.7.3. Thanks!
Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8
Hi, it would be cool if somebody could open a PR at https://bugs.freebsd.org/ I personally don't use FreeBSD 11 for any of my HAProxy-installations (yet), so I'm not really affected (yet) - but thanks for the heads-up. Regards, Rainer
Re: HaProxy Hang
> Am 03.03.2017 um 15:07 schrieb David King: > > Hi All > > Hoping someone will be able to help, we're running a bit of an interesting > setup > > we have 3 HAProxy nodes running freebsd 11.0 , each host runs 4 jails, each > running haproxy, but only one of the jails is under any real load > > Do you use ZFS? We have an internal software (some sort of monitoring agent) that also hangs in jails, from time to time. The guy who wrote it found out it’s because of mmap (I don’t know the specifics). The processes end up as unkillable in „D“ state and we need to reboot the hosts to fix it. As the purpose of the hosts is not to run the agent, we usually let it hang and restart when it’s convenient. The systems are FreeBSD 10.3, though (running nginx and varnish in different jails).
SNI healthcheck on backend?
Hi, I came upon this thread: http://discourse.haproxy.org/t/can-1-6-do-sni-on-backend/278/12 Is this true? I can't do healthchecks on a backend that needs to do SNI to the target-server? BTW: is there a (public) bug-database for haproxy? I couldn't find one on the web-page. As haproxy doesn't (seem to) use github, the authors might think about installing gitlab. Best Regards Rainer
Re: WAF in HAProxy
> Am 06.05.2016 um 00:15 schrieb Thierry FOURNIER >: > > Hi, > > You can look here: > > http://discourse.haproxy.org/t/ironbee-in-haproxy/92 > > Thierry > > Is that project actually alive? The last (and what looks like only) commit this year was to adjust the year for the copyright. That in general is not really the most assuring sign for a healthy open source project.
Re: Linux or FreeBSD ?
> Am 30.09.2015 um 16:25 schrieb Jeff Palmer: > > Arnall, > > > This advice is less of an haproxy specific response, and more of > general information. > > As someone who's tried to manage mixed infrastructure, I would push > back if possible, unles syour organization has decided to move to > freebsd entirely. > Very few do that. FreeBSD fulfills its purposes, though. Even if you try to standardize on one „flavor“ of Linux, you will still end up with other flavors - simply because not everything runs on your particular flavor. And you’re not going to run all of your applications on all of your platforms anyway. So the QA-effort should be manageable. But that doesn’t mean it’s wise to introduce a half dozen different platforms, either - unless you have enough people to handle all of it. How many systems (with Debian) are we talking about anyway? And how many HA-Proxies are supposed to be migrated? What are the sysadmin’s technical points for moving? Besides probably not wanting to deal with Debian’s head-ache-inducing idea of an OS - that’s a given ;-) Unless OP is doing some *really fancy stuff*, there’s IMO no pure technical show-stopper for a switch.
Re: Linux or FreeBSD ?
> Am 01.10.2015 um 01:22 schrieb Willy Tarreau: > >> > > I'd be tempted to place my judgement between yours and Jeff's. I'd say > that if the company is already using the target OS on any other place, > the cost of switching is low. If the load balancer is the opportunity > to introduce a new OS, it's a bad idea. By nature a load balancer is > very OS-dependant, and has bugs. Sometimes it's not trivial to tell > if a bug is in haproxy or the underlying OS until you get network > traces and/or strace output (BTW as far as I know, strace still doesn't > support amd64 on FreeBSD). Mixing the two can cast a bad image on the > new OS just because admins will initially not know well how to tune it > for the load and to ensure stability, will not easily troubleshoot > tricky issues, and a lot of frustration will result from this. > Probably. But OP’s admin will have his reasons for wanting FreeBSD in the picture. My guess would be that FreeBSD is the OS he’s more familiar with debugging. FreeBSD has ktrace - and dtrace (if you know how to use it, that is…) Here, most of our LBs run HAproxy on FreeBSD. Sometimes, they’re not. Because…reasons ;-) Why? Well, historically, most LBs and reverse-proxies ran FreeBSD (with NGINX). So it was more or less a „natural“ choice, with some pushing from my side (cough). FreeBSD has CARP. Linux has keepalived. etc. I don’t think we’ll ever get so much traffic that either one will be superior to the other. And I seriously doubt OP will. FreeBSD 10.1 has most of the optimizations that Netflix uses turned-on out of the box - but they do file-serving with NGINX. In their (extreme) case, it works better. Proxying/load-balancing is a bit different. I like FreeBSD because I can get a very stable, simple, low overhead, no-nonsense OS with a reasonable shelf-live and update-cycle while still being able to get up-to-date packages directly from upstream. > You should expect roughly the same performance on both OS so that is > not a consideration for switching or not switching. Really keep in > mind the admin cost, the cost of it being the exception in all your > system and possibly different debugging tools. It's very likely that > it will not be a problem, but better be aware of this. > That’s what you get by hiring a FreeBSD guy. If OP had hired a CentOS guy, I bet he'd want to switch everything to CentOS (or even Atomic Server…) ;-)
Re: Is FTP through haproxy at all viable?
I consider openssh for sftp pretty much unusable for clients/customers. I wouldn’t say that. Certainly true if they don’t actually know what they’re doing. As for the setup: yes, the first directory users can write to in a chroot-setup is a subdirectory of the home directory (because $HOME needs to be owned by root). But everything else is pretty simple. You don’t need any special devices or other stuff in the chroot itself. It basically just works in my experience. If you want to chroot a full, interactive shell, though, you’re jumping into a world of pain… Doesn’t have much to do with haproxy, though. Personally, I’m not sure if load-balancing FTP is worth the effort. Also, it looks like it’s quite „fragile“ and as such the load-balancing might break more often than a single-server without load-balancing.
Re: tcp-check for IMAP SSL ?
Am 01.01.2015 um 14:37 schrieb PiBa-NL piba.nl@gmail.com: Yosef Amir schreef op 1-1-2015 om 13:57: listen IMAP_SSL mode tcp bind :443 name VVM_SSL balance roundrobin tcp-check connect port 443 Maybe try the 'ssl' keyword as below. (i have not tested it at all..) tcp-check connect port 443 ssl option tcp-check tcp-check expect string ? server MIPS3 3.3.3.3 check server MIPS4 4.4.4.4 check Hi, Port 143 will actually be inline-TLS (STARTTLS). SSL is on port 993. The above answer should be correct, according to this: http://comments.gmane.org/gmane.comp.web.haproxy/19274 http://comments.gmane.org/gmane.comp.web.haproxy/19274 But only for SSL. Don’t know about inline-TLS. Rainer
Re: 1.5.9 crashes every 4 hours, like clockwork
Am 11.12.2014 um 15:26 schrieb David Adams dr...@yahoo.com mailto:dr...@yahoo.com: We are running 1.5.9 on Centos 6.5. It crashes 10 seconds (give or take a few seconds) after 1am, 5am, 9am, 1pm, 5pm and 9pm, like clockwork; let's call that CRASHTIME. Previously we'd been using 1.5.3 on the same hardware for some months without crashes. Once the crashes started we moved to 1.5.9 but they continue. If we manually restart it a minute or two before CRASHTIME it stills crashes when CRASHTIME arrives a minute or two later. Interesting. I’ve got a (single) VM where haproxy also crashes rather regularly (almost daily) at around 22:30-ish. I though it was because of 1.4.20-something, but it didn’t stop when I upgraded to 1.5.x Then, I thought it was FreeBSD 9 and upgraded to FreeBSD 10. It’s now on 10.1 and still crashes. Almost all my haproxy-VMs are actually provisioned with chef and are pretty similar and I’ve got this issue nowhere else. I build the package myself on my own poudriere-server and the same package works elsewhere on much busier servers without problems. We’ve got an icinga event-handler that restarts it… Rainer
HAPROXY for IMAP, SMTP
Hi, we use HAPROXY for incoming mail, outgoing mail (authenticated), POP3, IMAP. With incoming mail, I can make use of HAProxy’s send-proxy feature to make the source-IP known to the backend SMTP-servers. (Works in the lab, I just need to move a few hundred customers off port 25 for authenticated SMTP, as send-proxy is incompatible with authentication (right?)) But what about authenticated SMTP connections (which go on Port 587 or 465)?. We get a fair amount of abuse from hijacked accounts. I need to know the original IP from these connections, too, so I can quickly see if it connects from China, Pakistan or whatever (our customers are 99.99% only connecting from domestic fix and dynamic IPs and authenticated connections from multiple IPs from multiple countries to the same account are 100% hijacked). Same in principle for POP3 and IMAP. Is there no other way other than running TPROXY mode (which I want to avoid and is AFAIK also not recommended)? I have about 15k individual users. As traffic is going to be almost 100% encrypted in the near future, I can't even run something like SNORT on the LB and just process the logs from that…. Have the patches from this thread: http://marc.info/?t=13662203193r=1w=2 been incoporated into the HAproxy 1.5 source tree since then?
haproxy sending RSTs to backend-servers
Hi, I’ve configured nginx+haproxy in front of a couple of IIS servers. NGINX terminates SSL. configuration is as following: global log /var/run/log local5 log /var/run/log local1 notice #log loghostlocal0 info maxconn 4096 #debug #quiet user www group www daemon defaults log global modehttp retries 2 timeout client 50s timeout connect 5s timeout server 50s option dontlognull option forwardfor option httplog option redispatch balance leastconn http-check expect string server_up http-check disable-on-404 default-server minconn 50 maxconn 100 # Set up application listeners here. frontend app-main-prod mode http bind 0.0.0.0:8000 maxconn 2000 default_backend app-main-prod-back frontend app-import mode http bind 0.0.0.0:8001 maxconn 2000 default_backend app-import-back frontend app-images mode http bind 0.0.0.0:8002 maxconn 2000 default_backend app-images-back backend app-main-prod-back balance leastconn fullconn 2000 mode http option httpchk GET /healthcheck.aspx HTTP/1.1\r\nHost:\ www.app.ch\r\nConnection:\ close cookie SERVERID insert indirect nocache server appsrv-one 192.168.69.17:80 weight 1 maxconn 1000 check cookie s1 server appsrv-two 192.168.69.18:80 weight 1 maxconn 1000 check cookie s2 backend app-import-back balance leastconn fullconn 2000 mode http #option httpchk GET /healthcheck.aspx HTTP/1.1\r\nHost:\ import.app.ch\r\nConnection:\ close server appsrv-import-one 192.168.69.32:80 weight 1 maxconn 1000 check #server appsrv-import-two 192.168.69.33:80 weight 1 maxconn 1000 check backend app-images-back balance leastconn fullconn 2000 mode http option httpchk GET /healthcheck.aspx HTTP/1.1\r\nHost:\ images.app.ch\r\nConnection:\ close server appsrv-images-one 192.168.69.41:80 weight 1 maxconn 1000 check #server appsrv-images-two 192.168.69.42:80 weight 1 maxconn 1000 check listen admin 0.0.0.0:22002 mode http stats uri / What happens is that it will mostly work, but in wireshark, I see a lot of RST being sent from the haproxy-server to the backend IIS-servers. This doesn’t make sense and is probably the reason I see so many 50x in the logs and why occasionally gateway-errors are being shown to users because nginx can’t find any live servers… Can anyone see any obvious error in the config?
Is it possible to query the query the status of a server and use it in an ACL?
Hi, I want to take the status of a server of a given backend and use it in another backend or in the frontend. If that possible? I though there might be something simular to nbsrv() - but I haven't found anything. Best Regards Rainer
Can you balance-out service-checks better?
Hi, we will put haproxy in front of a Zimbra infrastructure (which we have split-up, so that there is a „front end“, with pop, imap, smtp and a „back end“, where the mail sits). I have too haproxy-servers (active/standby via CARP) that are checking the front-ends. I check: - smtp - smtps - submit - pop + pops - imap + imaps from both haproxy-servers simultaneously. If I use the default check frequency, it just bombards the servers with requests that often can’t even finish in the time it takes to launch the next check. If I increase the check-frequency too much, it will take longer to take a server out of the pool in case of failure - and checks still don’t „balance-out“ (or do they?). But they are all more or less connected: if one of them fails, it’s highly likely that all the others will fail, too. So, ideally, I’d like to have something like this: - check service A (maybe POP3) - wait maybe 30s - than check the next service (e.g. POP3S). - if one fails, remove that backend-server from the pool for all services - alternatively, instead of doing the above, re-schedule the checks so the next check happens immediately Does that sound insane? ;-)
Re: Can you balance-out service-checks better?
Am 28.08.2014 um 22:41 schrieb Baptiste bed...@gmail.com: Hi, maybe you could share your HAProxy configuration :) By default, HAProxy tests a service every 3s, which is fine. It just does a tcp connect, so nothing complicated for your server to handle. Since we switched to haproxy-1.5, I changed the checks to do a more or less full layer7-check (except for the SSL-services). Couldn’t get a match for the IMAP string it sends, so skipped that, too. Can you confirm that if POP fails on a server, it means that IMAP and SMTP will fail too? (this is what I'm understanding from your mail above). It’s very likely. All use the same backend-service in the end. There’s an additional pair of SMTP-servers here (ep01+ep02) - they are independent of the other two servers (pm01+pm02). But I’d also like to limit checking there, as of course all the checks for smtp, smtps+submit all go to the same postfix in the end…. Here’s the config. global log 127.0.0.1 local0 log 127.0.0.1 local1 notice #log loghostlocal0 info maxconn 4096 #debug #quiet user www group www daemon defaults log global modehttp retries 2 timeout client 50s timeout connect 5s timeout server 50s option dontlognull option forwardfor option httplog option redispatch balance roundrobin default-server minconn 50 maxconn 100 # Set up application listeners here. frontend pop3-pm mode tcp bind 192.168.185.254:110 maxconn 2000 default_backend pop3-pm-backend frontend imap4-pm mode tcp bind 192.168.185.254:143 maxconn 2000 default_backend imap4-pm-backend frontend pop3s-pm mode tcp bind 192.168.185.254:995 maxconn 2000 default_backend pop3s-pm-backend frontend imap4s-pm mode tcp bind 192.168.185.254:993 maxconn 2000 default_backend imap4s-pm-backend frontend smtp-ep mode tcp bind 192.168.185.254:25 maxconn 2000 default_backend smtp-ep-backend frontend smtps-ep mode tcp bind 192.168.185.254:465 maxconn 2000 default_backend smtps-ep-backend frontend submit-ep mode tcp bind 192.168.185.254:587 maxconn 2000 default_backend submit-ep-backend frontend smtp-zimbra mode tcp bind 192.168.185.253:25 maxconn 2000 default_backend smtp-zimbra-backend frontend http-webmail bind 192.168.185.254:5000 maxconn 6000 default_backend http-webmail-backend # # # backend pop3-pm-backend balance roundrobin mode tcp option tcp-check tcp-check expect string +OK\ POP3\ ready tcp-check send quit\r\n tcp-check expect string +OK server pm01 192.168.185.233:110 check inter 30s fastinter 2s downinter 2s server pm02 192.168.185.234:110 check inter 30s fastinter 2s downinter 2s backend pop3s-pm-backend balance roundrobin mode tcp # this is ssl, so it does not work here # option tcp-check # tcp-check expect string +OK\ POP3\ ready server pm01 192.168.185.233:995 check inter 30s fastinter 2s downinter 2s server pm02 192.168.185.234:995 check inter 30s fastinter 2s downinter 2s backend imap4-pm-backend balance roundrobin mode tcp option tcp-check tcp-check expect rstring OK\ IMAP4\ ready tcp-check send 001 logout\r\n #tcp-check expect string *\ BYE\ Zimbra\ IMAP\ server\ terminating\ connection\r\n001\ OK\ completed server pm01 192.168.185.233:143 check inter 30s fastinter 2s downinter 2s server pm02 192.168.185.234:143 check inter 30s fastinter 2s downinter 2s backend imap4s-pm-backend balance roundrobin mode tcp server pm01 192.168.185.233:993 check inter 30s fastinter 2s downinter 2s server pm02 192.168.185.234:993 check inter 30s fastinter 2s downinter 2s backend smtp-ep-backend balance roundrobin mode tcp option smtpchk HELO mail.this.here server ep01 192.168.185.198:25 check inter 45s fastinter 2s downinter 2s server ep02 192.168.185.199:25 check inter 45s fastinter 2s downinter 2s backend smtps-ep-backend balance roundrobin mode tcp #option smtpchk HELO mail.this.here server ep01 192.168.185.198:465 check inter 45s fastinter 2s downinter 2s server ep02 192.168.185.199:465 check inter 45s fastinter 2s downinter 2s backend submit-ep-backend balance roundrobin mode tcp option smtpchk HELO mail.scalera.ch server ep01 192.168.185.198:587 check inter 45s fastinter 2s downinter 2s server ep02 192.168.185.199:587 check inter 45s fastinter 2s downinter 2s backend smtp-zimbra-backend balance roundrobin mode tcp option smtpchk HELO mail.this.here server pm01 192.168.185.233:25 check inter 45s fastinter 2s downinter 2s server pm02 192.168.185.234:25 check inter 45s fastinter 2s downinter 2s backend http-webmail-backend balance leastconn mode http option httpchk GET / #http-check expect string Webmail Login Page http-check expect string Web Client Login Page http-check disable-on-404 cookie SERVERID insert indirect nocache server pm1 192.168.185.233:80 check maxconn 3000 cookie s1 inter 10s fastinter 2s downinter
Re: Can you balance-out service-checks better?
Am 28.08.2014 um 23:21 schrieb Baptiste bed...@gmail.com: Ok, I would create a monitoring backend, such as below: Hey, thanks a lot! I will try this and report back. Best Regards, Rainer
Re: Load balancing FTP with HAProxy behind a firewall
hdr(host) ACL only applies to HTTP. Furthermore, I'm not sure there is a notion of Host header in FTP ;) Last time I looked (admittedly with 1.4) into FTP+HAProxy, the end-result was that it was just not possible. AFAIK, you can use LVS for that on Linux.
Re: haproxy dumps core
Am 30.07.2013 um 21:40 schrieb Lukas Tribus luky...@hotmail.com: Hi Rainer! I'm using haproxy on FreeBSD 9.1-amd64 inside a VMware VM. I realized that when I have a situation where all servers in a backend are down, haproxy crashes: Jul 30 08:03:52 px2-bla kernel: pid 58816 (haproxy), uid 80: exited on signal 11 (core dumped) pkg info|grep haproxy haproxy-1.4.24 The Reliable, High Performance can you post the output of haproxy -vv? After some tinkering, I got a core-dump out of it: The core-dump doesn't look very useful, seems like the debugging symbols where stripped. Hi, sorry, I haven't had time to look into this, but now I've been able to generate a core (and run it through gdb) gdb /usr/local/sbin/haproxy haproxy.3272 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as amd64-marcel-freebsd... Core was generated by `haproxy'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libcrypt.so.5...done. Loaded symbols for /lib/libcrypt.so.5 Reading symbols from /usr/local/lib/libpcreposix.so.0...done. Loaded symbols for /usr/local/lib/libpcreposix.so.0 Reading symbols from /usr/local/lib/libpcre.so.3...done. Loaded symbols for /usr/local/lib/libpcre.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x0043d0e9 in process_session (t=0x801866f00) at src/session.c:1434 1434src/session.c: No such file or directory. in src/session.c [New Thread 801807400 (LWP 100105/unknown)] [New LWP 100114] (gdb) bt #0 0x0043d0e9 in process_session (t=0x801866f00) at src/session.c:1434 #1 0x00408420 in process_runnable_tasks (next=0x7fffdafc) at src/task.c:234 #2 0x004028e3 in run_poll_loop () at src/haproxy.c:1002 #3 0x0040455d in main (argc=value optimized out, argv=0x7fffdba0) at src/haproxy.c:1288 Can you make something of this? I found it may be a config-file problem. Apart from comments, the only difference between a config-file that makes haproxy dump core and one that doesn't is: maxconn 500 server server1 ip:80 weight 1 check --- maxconn 500 server server1 ip:80 weight 1 check Best Regards Rainer
haproxy dumps core
8000 #option httpchk GET /ip_monitor_mysql.php HTTP/1.1\r\nHost: www.s.domain\r\nConnection:\ close server app2 first.ip:80 weight 1 check server input1 second.ip:80 weight 1 check backend servers-old-p fullconn 8000 #option httpchk GET /ip_monitor_mysql.php HTTP/1.1\r\nHost: www.p.domain\r\nConnection:\ close server app2 first.ip:80 weight 1 check server input1 second.ip:80 weight 1 check backend servers-old-s-stage fullconn 8000 #option httpchk GET /ip_monitor_mysql.php HTTP/1.1\r\nHost: s-stage.1st.domain\r\nConnection:\ close server app2 first.ip:80 weight 1 check server input1 second.ip:80 weight 1 check backend servers-old-p-stage fullconn 8000 #option httpchk GET /ip_monitor_mysql.php HTTP/1.1\r\nHost: p-stage.1st.domain\r\nConnection:\ close server app2 first.ip:80 weight 1 check server input1 second.ip:80 weight 1 check listen admin 0.0.0.0:22002 mode http stats uri / Regards, Rainer
Re: haproxy dumps core
Am Tue, 30 Jul 2013 21:40:34 +0200 schrieb Lukas Tribus luky...@hotmail.com: Hi Rainer! I'm using haproxy on FreeBSD 9.1-amd64 inside a VMware VM. I realized that when I have a situation where all servers in a backend are down, haproxy crashes: Jul 30 08:03:52 px2-bla kernel: pid 58816 (haproxy), uid 80: exited on signal 11 (core dumped) pkg info|grep haproxy haproxy-1.4.24 The Reliable, High Performance can you post the output of haproxy -vv? (px2-bla /root) 0 # haproxy -vv HA-Proxy version 1.4.24 2013/06/17 Copyright 2000-2013 Willy Tarreau w...@1wt.eu Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -O2 -pipe -fno-strict-aliasing -DFREEBSD_PORTS OPTIONS = USE_STATIC_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Available polling systems : kqueue : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use kqueue. After some tinkering, I got a core-dump out of it: The core-dump doesn't look very useful, seems like the debugging symbols where stripped. Could you recompile haproxy with the following CFLAGS: make CFLAGS=-g -O0 TARGET=[...] and regenerate the core-dump. The GDB output should be more informative then. If the executable comes from a packaging system (ports?), you may be able to use a debug-package instead of recompiling haproxy (although compiler optimization may obfuscate the backtrace). I'll look into it. It's created by our poudriere package-building system. Regards, Rainer
Re: Empty IP when forwardfor enabled
On Monday 19 January 2009 03:19:51 Rodrigo wrote: Hi, I've setup two frontends balanced with Haproxy 1.3.15.7 (excellent software) in another server: Load balancer / \ /\ / \ Frontend 1Frontend 2 I have enabled option forwardfor. I have set LogFormat in Apache configuration file as follows: LogFormat %{X-Forwarded-For}i %l %u %t \%r\ %s %b \%{Referer}i\ \%{User-Agent}i\ combined but many times the IP field appears empty. I've read on this mailling list that it has something to do with KeepAliveTimeout on Apache. I rised up it from 6 to 15, but no luck. How could I fix this? You probably need option httpclose. -Rainer