Re: haproxy issue
Hi, On Wed, Oct 15, pasquale scalea wrote: i have configure haproxy-1.5.4-1.el6.x86_64 with this config ... backend view_https mode tcp option tcplog option httpchk OPTIONS / http-check expect status 200 option ssl-hello-chk #make sure we can talk SSL, not just TCP balance source balance roundrobin stick on src stick-table type ip size 100m expire 1h server milap9900 172.16.74.17:443 check id 1 inter 10s rise 5 fall 2 server milap9901 172.16.74.18:443 check id 2 inter 10s rise 5 fall 2 How are you trying to do the server health checks ? Are you trying to use OPTIONS / over ssl connection or just ssl-hello-chk ? If you're trying to do OPTIONS / with ssl then maybe something like this might work: # disable option ssl-hello-chk option httpchk OPTIONS / server milap9900 172.16.74.17:443 check ssl id 1 inter 10s rise 5 fall 2 server milap9901 172.16.74.18:443 check ssl id 2 inter 10s rise 5 fall 2 (and make sure you've compiled haproxy with ssl support: haproxy -vv) (and you need to configure ca certs or use verify none/ssl-server-verify none) http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#5.2-ssl or http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#5.2-check-ssl Also check that the backend server responds to OPTIONS / with (for example with: openssl s_client -connect 172.16.74.18:443 and type OPTIONS / HTTP/1.0 and hit enter twice) Also are you trying to send the same client to the same server: balance source / balance roundrobin / stick on src ? http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-balance http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4.2-stick%20on Then balance source or balance roundrobin + stick on src should do the trick I don't think you'll need all three. Cheers, -Jarno -- Jarno Huuskonen
Re: HAProxy proxying JDBC requests two a percona cluster
Hi, On Wed, Oct 15, Dennis Plöger | getit GmbH wrote: Hi! Hmm... I just understood, that we're probably talking about different things here. As you can see in my configuration, I haven't specify a maxconn value. From the configuration documentation I didn't understand, what the default value is. I just updated my configuration based on this tutorial and will try again: http://www.percona.com/doc/percona-xtradb-cluster/5.5/howtos/haproxy.html Same error (nothing in the logs however). My haproxy.cfg now looks like this: option dontlognull Are you using any kind of connection pool on hybis/jdbc ? (Hybris opens/keeps open multiple connections and reuses the open connection). Does the connection pool do any connection checking (periodically checks that the connection is still alive) ? (Maybe pool opens connections that sit idle for too long - haproxy closes idle connection (50s?)) Maybe try with no option dontlognull ? Are all connections in logs logged with normal -- session state ? http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#8.5 -Jarno retries 3 option redispatch maxconn 2000 timeout connect 5 timeout client 5 timeout server 5 listen mysql-cluster 127.0.0.1:3306 mode tcp balance roundrobin option httpchk server shopclusterdb1 10.10.42.82:3306 check port 9200 inter 12000 rise 3 fall 3 server shopclusterdb2 10.10.42.83:3306 check port 9200 inter 12000 rise 3 fall 3 backup Kind regards Dennis -- Dennis Plöger Systemadministrator und Berater -- Besuchen Sie uns auf unseren nächsten Veranstaltungen: - konaktiva, 11.11.2014, Halle 3 B / Stand G4 (Westfalenhallen), Dortmund -- getit - Gesellschaft für Technologie- und Informationstransfer mbH Emil-Figge-Straße 76-80 44227 Dortmund Tel: +49.231.9742.355 Fax: +49.231.9742.356 http://www.getit.de Amtsgericht Dortmund, HRB-Nr. 26548 Geschäftsführung: Dr. Thomas Krämerkämper (Vorsitzender) Christoph Hecker -- Jarno Huuskonen
RE: Understanding haproxy cpu usage in ssl-offload scenario
Hi Dennis, Now from what I read on the net about haproxy I expected that the load-balancing isn't that cpu intensive and that most of the cpu usage should be generated by the ssl offloading but what I'm seeing is that the two cores 0 and 4 are at 20% idle while the rest is at 80-90% idle and that cpu core 4 shows 20% soft interrupt usage while all other cores show 0% soft interrupt usage. The statistics page also shows an idle time of only 20%. This sounds to me like a kernel thing. What exact kernel are you using? If you are on CentOs6, you should give CentOs 7 a try (which has a 3.10 kernel). There is probably some low level kernel/irq tuning that could help in this situation as well, but I never did it. Regards, Lukas
Stick-table misconfiguration; haproxy should throw error
I just wanted to report an issue that had me hitting my head on the table for a few hours: I reconfigured a development environment to use Chef to generate haproxy config files. Every 2-3 minutes or so, depending on load, haproxy would crash with a `trap divide error` (divide by zero). It worked fine previously, and during those few minutes it works perfectly, but the trap divide error persisted. Tried a new machine, still got it. Tried every 1.5 version from 1.5.5 all the way to the early dev builds, still got it. Turns out the culprit was a misnamed variable on my end that was causing my stick table definition to read `stick-table type ip ,bytes-in-rate(),http_err_rate(10s)`. That is, there was no period defined on the bytes-in-rate. It would be very nice if haproxy would regard this as a configuration error and throw on start rather than crash minutes later with a cryptic trap divide error. Thanks for a great project. Best, Sam signature.asc Description: Message signed with OpenPGP using GPGMail
Question about srv_conn and beconn
Hi haproxy team: I see the document about these: beconn is the total number of concurrent connections handled by the backend when the session was logged. srv_conn is the total number of concurrent connections still active on the server when the session was logged. fell confused about these two log values, what's the different between them ? and are these two values is at the begin process time values or at the end process doing log values? Regards
Re: HAProxy proxying JDBC requests two a percona cluster
Hi! Maybe try with no option dontlognull ? Are all connections in logs logged with normal -- session state ? http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#8.5 Oh. Thanks for mentioning that page (I somehow never got there). I now saw, that the connections are dropped with cD - meaning, that HAproxy thought, the server would be down. Could that be something in conjunction with the Percona-check on port 9200? Kind regards Dennis -- Dennis Plöger Systemadministrator und Berater -- Besuchen Sie uns auf unseren nächsten Veranstaltungen: - konaktiva, 11.11.2014, Halle 3 B / Stand G4 (Westfalenhallen), Dortmund -- getit - Gesellschaft für Technologie- und Informationstransfer mbH Emil-Figge-Straße 76-80 44227 Dortmund Tel: +49.231.9742.355 Fax: +49.231.9742.356 http://www.getit.de Amtsgericht Dortmund, HRB-Nr. 26548 Geschäftsführung: Dr. Thomas Krämerkämper (Vorsitzender) Christoph Hecker
Re: HAProxy proxying JDBC requests two a percona cluster
Hello! Oh. Thanks for mentioning that page (I somehow never got there). I now saw, that the connections are dropped with cD - meaning, that HAproxy thought, the server would be down. Getting back on that. I understand, that D means that HAProxy thinks, the server is down. Shouldn't hatop show something then that indicates a downtime on the server? Because hatop shows nothing in that direction. All's up, no downtime. Kind regards Dennis -- Dennis Plöger Systemadministrator und Berater -- Besuchen Sie uns auf unseren nächsten Veranstaltungen: - konaktiva, 11.11.2014, Halle 3 B / Stand G4 (Westfalenhallen), Dortmund -- getit - Gesellschaft für Technologie- und Informationstransfer mbH Emil-Figge-Straße 76-80 44227 Dortmund Tel: +49.231.9742.355 Fax: +49.231.9742.356 http://www.getit.de Amtsgericht Dortmund, HRB-Nr. 26548 Geschäftsführung: Dr. Thomas Krämerkämper (Vorsitzender) Christoph Hecker
Re: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hello Lukas, Thanks for your reply. You can find my additional information in your text below. Am 17.10.2014 um 01:32 schrieb Lukas Tribus luky...@hotmail.com: Gonna need to see your configuration to be able to help you, especially ssl and http related parts. ##- # Global settings #- global log 127.0.0.1 local2 debug alert chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4096 userhaproxy group haproxy daemon nbproc 1 #ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA #- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #- defaults modehttp log global option httplog option logasap option redispatch balance roundrobin timeout connect 1 # default 10 second time out if a backend is not found timeout client 8640 timeout server 30 timeout queue 5000 maxconn 6 retries 3 default_backend deny_backend #- # test configuration #- frontend test1 bind *:443 ssl crt /etc/pki/tls/certs/domain.com-haproxy.pem #ciphers HIGH:RC4-SHA:!ADH #ciphers AES:RC4:ALL:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:!ECDH reqadd X-Forwarded-Proto:\ https acl nourlurl / acl baseurl url /java-app acl is_websocket hdr(Upgrade) -i WebSocket acl is_websocket path_beg /java-app/r3 acl is_download path_beg /java-app/ acl admin_urlpath_beg /admin acl is_admin src 192.168.1.0/24 redirect location http://www.domain.com/ if nourl redirect location https://test.domain.com/java-app/ if baseurl use_backend test1_socket_backend if is_websocket use_backend test1_website_backend if is_download ! is_websocket use_backend private_monitoring if is_admin admin_url #-Backends backend deny_backend option httpclose reqideny .* backend private_monitoring stats enable stats uri /admin?stats stats refresh 10s # Test1 backend test1_website_backend option httpclose server test1 internal hostname:8080 backend test1_socket_backend no option httpclose server test1 internal hostname:8080 #- — changing bind *:443 ssl crt /etc/pki/tls/certs/domain.com-haproxy.pem to bind *:81 (and removing the https line next to it) makes it run with stunnel. Out of the back of my mind I recall Java has problems with DHE cihpers when the dh size is more than 1024 bits. Could that be your case? I read about it anywhere and was aware of it (a known problem with Java 7). I tested with the default of 1024 bit but also tried with other values. There was noch change in behavior. Can you share a tcpdump capture of the failed handshake (don't forget -s0 otherwise packets will be truncated). 43 29.413080 CLIENT HAPROXYTCP 80 62199→443 [SYN] Seq=0 Win=65535 Len=0 MSS=1452 WS=16 TSval=666771533 TSecr=0 SACK_PERM=1 44 29.413184 HAPROXYCLIENT TCP 76 443→62199 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1791798027 TSecr=666771533 WS=64 45 29.430494 CLIENT HAPROXYTCP 68 62199→443 [ACK] Seq=1 Ack=1 Win=132480 Len=0 TSval=666771557 TSecr=1791798027 46 29.435106 CLIENT HAPROXYSSLv2213Client Hello 47 29.440451 HAPROXYCLIENT TLSv1.2 1508 Server Hello 48 29.440473 HAPROXYCLIENT TLSv1.2 1134 Certificate 49 29.461553 CLIENT HAPROXYTCP 68 62199→443 [ACK]
RE: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hi Heiko, #- # test configuration #- frontend test1 bind *:443 ssl crt /etc/pki/tls/certs/domain.com-haproxy.pem #ciphers HIGH:RC4-SHA:!ADH #ciphers AES:RC4:ALL:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:!ECDH reqadd X-Forwarded-Proto:\ https acl nourl url / acl baseurl url /java-app acl is_websocket hdr(Upgrade) -i WebSocket acl is_websocket path_beg /java-app/r3 acl is_download path_beg /java-app/ acl admin_url path_beg /admin acl is_admin src 192.168.1.0/24 redirect location http://www.domain.com/ if nourl redirect location https://test.domain.com/java-app/ if baseurl use_backend test1_socket_backend if is_websocket use_backend test1_website_backend if is_download ! is_websocket use_backend private_monitoring if is_admin admin_url #-Backends backend deny_backend option httpclose reqideny .* backend private_monitoring stats enable stats uri /admin?stats stats refresh 10s # Test1 backend test1_website_backend option httpclose server test1 internal hostname:8080 backend test1_socket_backend no option httpclose Better switch to option http-tunnel here. Out of the back of my mind I recall Java has problems with DHE cihpers when the dh size is more than 1024 bits. Could that be your case? I read about it anywhere and was aware of it (a known problem with Java 7). I tested with the default of 1024 bit but also tried with other values. There was no change in behavior. How exactly did you modify this value? Do you have those dh-params in your pem file? 46 29.435106 CLIENT HAPROXY SSLv2 213 Client Hello 47 29.440451 HAPROXY CLIENT TLSv1.2 1508 Server Hello 48 29.440473 HAPROXY CLIENT TLSv1.2 1134 Certificate Can you try force-tlsv10 on the bind line, to see if that changes anything? If that doesn't help, can you send me the full tcpdump capture file offlist (tcpdump -ns0 -w capture.cap tcp port 443 or something like that)? Do you have any way to enable the java debug console and see what it says? Regards, Lukas
RE: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Used the bind parameter before which did / does not help and created a tcpdump with the mentioned settings (DH = 1024 Bit and force tls) with your requested parameters. Something doesn't add up. The handshake you sent me is still negotiating TLSv1.2 and TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027), but: - Java 7 doesn't support TLSv1.2 - Java 7 doesn't support this cipher - HAProxy with force-tlsv10 is supposed to force TLSv1.0 Are you sure this is a Java 7 JRE connecting? Are you sure forcetls is configured and HAProxy has been properly restarted? Please also provide the output of haproxy -vv. Thanks, Lukas
Re: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hello Lukas, Am 17.10.2014 um 18:11 schrieb Lukas Tribus luky...@hotmail.com: Used the bind parameter before which did / does not help and created a tcpdump with the mentioned settings (DH = 1024 Bit and force tls) with your requested parameters. Something doesn't add up. The handshake you sent me is still negotiating TLSv1.2 and TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027), but: - Java 7 doesn't support TLSv1.2 - Java 7 doesn't support this cipher - HAProxy with force-tlsv10 is supposed to force TLSv1.0 Are you sure this is a Java 7 JRE connecting? Are you sure forcetls is configured and HAProxy has been properly restarted? The client used for the test has Oracle Java 1.7.0_72 (JDK) installed on Mac OS X 10.9. The bind is done this way: — bind *:443 ssl crt /etc/pki/tls/certs/domain-haproxy.pem force-tlsv10 — Currently, I start haproxy manually with this command (in the same shell I edit the config file, thus I have to stop haproxy with CTRL-C for changes): — haproxy -d -f /etc/haproxy/haproxy.cfg — Please also provide the output of haproxy -vv. HA-Proxy version 1.5.5 2014/10/07 Copyright 2000-2014 Willy Tarreau w...@1wt.eu Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.3 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Thanks again and best regards, Heiko --- Heiko Burghardt IT Infrastructure -- .. Riege Software International GmbH Phone: +49 2159 91480 Mollsfeld 10 Fax: +49 2159 914811 40670 MeerbuschWeb: www.riege.com GermanyE-Mail: burgha...@riege.com -- -- Commercial Register: Managing Directors: Amtsgericht Neuss HRB-NR 4207 Christian Riege VAT Reg No.: DE120585842 Gabriele Riege Johannes Riege TobiasRiege .. YOU CARE FOR FREIGHT, WE CARE FOR YOU
RE: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hi Heiko, Currently, I start haproxy manually with this command (in the same shell I edit the config file, thus I have to stop haproxy with CTRL-C for changes): — haproxy -d -f /etc/haproxy/haproxy.cfg — I see. Can you run it through strace -tt, Not that I expect to see why the TLS handshake fails, just to confirm that its indeed haproxy that accepts the connection (just prepend your command above with strace -tt). Attach the strace output to a txt file to the mail, as it will be long. Also, please try the bind keywords no-tlsv12, no-tlsv11 and ciphers TLS_RSA_WITH_RC4_128_SHA. If this makes it work, please apply the attached debug patch and just run it with force-tlsv10, I would like to know if that call fails. Anyway, I still think we don't see the whole picture. You don't have any SSL/TLS intercepting middleboxes between your client and your server, correct? Regards, Lukas debugforcetlsv10.patch Description: Binary data
Re: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hello Lukas,
Thanks for your update.
Currently, I start haproxy manually with this command (in the same shell I
edit the config file, thus I have to stop haproxy with CTRL-C for changes):
—
haproxy -d -f /etc/haproxy/haproxy.cfg
—
I see. Can you run it through strace -tt, Not that I expect to see why the TLS
handshake fails, just to confirm that its indeed haproxy that accepts the
connection (just prepend your command above with strace -tt). Attach the
strace output to a txt file to the mail, as it will be long.
I will send the entire strace output separately to you as it very long but the
short form for the mailing list:
—
00:15:55.786195 gettimeofday({1413584155, 786244}, NULL) = 0
00:15:55.786280 accept4(4, {sa_family=AF_INET, sin_port=htons(57248),
sin_addr=inet_addr(„MY IPV4 ADDRESS ON INTERNET)}, [16], SOCK_NONBLOCK) = 6
00:15:55.786548 accept4(4, 0x7fff2fbe9880, [128], SOCK_NONBLOCK) = -1 EAGAIN
(Resource temporarily unavailable)
00:15:55.786629 gettimeofday({1413584155, 786643}, NULL) = 0
00:15:55.786772 read(6, 0x119b343, 5) = -1 EAGAIN (Resource temporarily
unavailable)
00:15:55.786890 epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLIN|0x2000, {u32=6,
u64=6}}) = 0
00:15:55.786943 gettimeofday({1413584155, 786956}, NULL) = 0
00:15:55.786983 epoll_wait(3, {{EPOLLIN, {u32=6, u64=6}}}, 200, 1000) = 1
00:15:55.815985 gettimeofday({1413584155, 816008}, NULL) = 0
00:15:55.816038 gettimeofday({1413584155, 816051}, NULL) = 0
00:15:55.816092 read(6, \200\217\1\3\3, 5) = 5
00:15:55.816154 recvfrom(6, NULL, 2147483647,
MSG_TRUNC|MSG_DONTWAIT|MSG_NOSIGNAL, NULL, NULL) = 140
00:15:55.816208 recvfrom(6, 0, 2147483647, 16480, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
00:15:55.816312 sendto(5, 150Oct 18 00:15:55 haproxy[100..., 116,
MSG_DONTWAIT|MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(514),
sin_addr=inet_addr(127.0.0.1)}, 16) = 116
00:15:55.816565 close(6)= 0
—
This is the connect to haproxy where the web socket upgrade shall take place.
Also, please try the bind keywords no-tlsv12, no-tlsv11 and
ciphers TLS_RSA_WITH_RC4_128_SHA. If this makes it work, please apply
the attached debug patch and just run it with force-tlsv10, I would like
to know if that call fails.
I added the parameters except TLS_RSA_WITH_RC4_128_SHA as it is not available
in openssl. This one seems to be the equivalent here: RC4-SHA
Thus, my bind looks like this:
bind *:443 ssl crt /etc/pki/tls/certs/domain-haproxy.pem force-tlsv10 no-tlsv12
no-tlsv11 ciphers RC4-SHA
But again, the web socket upgrade does not take place. haproxy log still says:
Connection closed during SSL handshake.
Am I right that the patch is only useful for you on a positive result?
Anyway, I still think we don't see the whole picture. You don't have
any SSL/TLS intercepting middleboxes between your client and your
server, correct?
All my tests were established this way:
Java client - Internet router doing SNAT to Internet IP - INTERNET -
Company firewall doing DNAT to internal, private IP - haproxy server
I always re-checked the company firewall but this connection is not logged in
any way.
Best regards,
Heiko
