Re: HAProxy 1.5.10 on FreeBSD 9.3 - status page questions
Hey guys, thank you very much for the details. We moved from a simple perl script that produced new connections all the time for pushing the data onto the KAFKA message queue to an intelligent producer written in Erlang some months ago. This producer keeps the connections alive and therefore the statistics do not get updated: [deploy_user@haproxy ~] echo 'show sess' | sudo socat stdio /var/run/haproxy.sock 0x801885800: proto=tcpv4 src=10.5.1.161:12538 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=878202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=1,ex=] s1=[7,8h,fd=36,ex=] exp= 0x801885400: proto=tcpv4 src=10.5.1.161:14709 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=2,ex=] s1=[7,8h,fd=37,ex=] exp= 0x801885c00: proto=tcpv4 src=10.5.1.161:24722 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=3,ex=] s1=[7,8h,fd=38,ex=] exp= 0x801886000: proto=tcpv4 src=10.5.1.161:40036 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=8,ex=] s1=[7,8h,fd=39,ex=] exp= 0x801886400: proto=tcpv4 src=10.5.1.161:36941 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=9,ex=] s1=[7,8h,fd=40,ex=] exp= 0x801886800: proto=tcpv4 src=10.5.1.161:25521 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=878202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=10,ex=] s1=[7,8h,fd=41,ex=] exp= 0x801886c00: proto=tcpv4 src=10.5.1.161:43885 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=11,ex=] s1=[7,8h,fd=42,ex=] exp= 0x801887000: proto=tcpv4 src=10.5.1.161:42777 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=12,ex=] s1=[7,8h,fd=43,ex=] exp= 0x801887400: proto=tcpv4 src=10.5.1.161:47500 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=13,ex=] s1=[7,8h,fd=44,ex=] exp= 0x801887800: proto=tcpv4 src=10.5.1.161:56415 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=14,ex=] s1=[7,8h,fd=45,ex=] exp= 0x801887c00: proto=tcpv4 src=10.5.1.161:39029 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=15,ex=] s1=[7,8h,fd=46,ex=] exp= 0x801888000: proto=tcpv4 src=10.5.1.161:14409 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=16,ex=] s1=[7,8h,fd=47,ex=] exp= 0x801888400: proto=tcpv4 src=10.5.1.161:10419 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=17,ex=] s1=[7,8h,fd=48,ex=] exp= 0x80100: proto=tcpv4 src=10.5.1.161:58710 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=808000h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=18,ex=] s1=[7,8h,fd=49,ex=] exp= 0x801888c00: proto=tcpv4 src=10.5.1.161:59694 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=19,ex=] s1=[7,8h,fd=50,ex=] exp= 0x801889000: proto=tcpv4 src=10.5.1.161:12557 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=20,ex=] s1=[7,8h,fd=51,ex=] exp= 0x801889400: proto=tcpv4 src=10.5.1.161:49539 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=21,ex=] s1=[7,8h,fd=52,ex=] exp= 0x801889800: proto=tcpv4 src=10.5.1.161:12768 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=22,ex=] s1=[7,8h,fd=53,ex=] exp= 0x801889c00: proto=tcpv4 src=10.5.1.161:42780 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=] rp[f=008000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=23,ex=] s1=[7,8h,fd=54,ex=] exp= 0x80188a000: proto=tcpv4 src=10.5.1.161:23841 fe=KAFKA be=KAFKA_BACKEND srv=KAFKA_PRIMARY ts=08 age=9d22h calls=2 rq[f=848202h,i=0,an=00h,rx=,wx=,ax=]
Re: HAProxy 1.5.10 on FreeBSD 9.3 - status page questions
Hi Tobias, On Fri, Feb 20, 2015 at 03:58:10PM +0100, Tobias Feldhaus wrote: Hey guys, thank you very much for the details. We moved from a simple perl script that produced new connections all the time for pushing the data onto the KAFKA message queue to an intelligent producer written in Erlang some months ago. This producer keeps the connections alive and therefore the statistics do not get updated: (...) OK thanks for confirming this was the cause. Out of curiosity, how often does your producer write to the socket, how much data each time, etc ? I'm asking because I think that the loss of the continuous stat often is sad in situations like yours, and at the same time I'm thinking that maybe we could managed to force *some* of the transfers to wake the session up to update the counters once in a while. The principle could be based on volume, number of reads or anything that we already have, we just have to find the correct metric that serves the purpose well and that doesn't affect performance. Thanks, Willy
NOSRV/BADREQ from some Java based clients
We have been in the process of deploying HAProxy as a SSL terminator between our client software and back end services. In the testing phases, everything is working great and looking good with one exception: Some old client software that utilizes a Java SSL implementation fail to connect and we end up with logs like this: [20/Feb/2015:15:49:51.632] https_frontend~ https_frontend/NOSRV -1/-1/-1/-1/23 400 187 - - CR-- 0/0/0/0/0 0/0 BADREQ Without HAProxy in the mix, these same clients connect up to our Mochiweb services (via SSL) just fine. Additionally, our newer clients that are OpenSSL based communicate with HAProxy (termination) - Mochiweb (via HTTP) just fine as well. From what I can tell, it appears as though we may have a combination of two bad things: 1) Clients sending some sort of non-standard handshake 3) Mochiweb has been allowing it. Some additional gritty details: * socat 'show errors' shows 0 errors * The same bad clients fail to connect to a OpenSSL s_server (logs below) Since we can't even properly connect to s_server, that may be the end of the road for those clients. However, I'm hoping there may be something that could be configured to allow them through HAProxy. Below is a s_server log. Note the read failure at the end. A similar capture in the view of Wireshark is below that. Lastly, *with* HAProxy when the NOSRV/BADREQ is issued, the client is sent a encrypted 400 Bad Request. Any help/tips appreciated! This represents a large client base that unfortunately cannot be updated for the time being. If we cannot go through HAProxy directly, the next step is to figure out a way to route old clients around it :( --/snip/-- sudo openssl s_server -accept 443 -cert ~/Downloads/json_rpc_server_cert_and_key.pem -msg -debug -state Using default temp DH parameters Using default temp ECDH parameters ACCEPT SSL_accept:before/accept initialization read from 0x1a43e90 [0x1a49580] (11 bytes = 11 (0xB)) - 16 03 01 00 d3 01 00 00-cf 03 01 ... read from 0x1a43e90 [0x1a4958e] (205 bytes = 205 (0xCD)) - 54 e7 c3 80 5c a7 15 6b-ac 69 3e 5f b2 9e ba 87 T...\..k.i_ 0010 - 53 19 92 5b 0a 21 e5 32-f7 29 22 8e 03 0c 54 f4 S..[.!.2.)...T. 0020 - 20 87 17 d7 e9 44 c6 cc-76 2e c0 aa 54 05 94 afD..v...T... 0030 - 9c f1 24 59 ac fb 6b 7c-c0 7e 0b b8 65 f8 48 a5 ..$Y..k|.~..e.H. 0040 - fc 00 46 00 04 00 05 00-2f 00 35 c0 02 c0 04 c0 ..F./.5. 0050 - 05 c0 0c c0 0e c0 0f c0-07 c0 09 c0 0a c0 11 c0 0060 - 13 c0 14 00 33 00 39 00-32 00 38 00 0a c0 03 c0 3.9.2.8. 0070 - 0d c0 08 c0 12 00 16 00-13 00 09 00 15 00 12 00 0080 - 03 00 08 00 14 00 11 00-ff 01 00 00 40 00 0b 00 @... 0090 - 04 03 00 01 02 00 0a 00-34 00 32 00 0e 00 0d 00 4.2. 00a0 - 19 00 0b 00 0c 00 18 00-09 00 0a 00 16 00 17 00 00b0 - 08 00 06 00 07 00 14 00-15 00 04 00 05 00 12 00 00c0 - 13 00 01 00 02 00 03 00-0f 00 10 00 11. TLS 1.0 Handshake [length 00d3], ClientHello 01 00 00 cf 03 01 54 e7 c3 80 5c a7 15 6b ac 69 3e 5f b2 9e ba 87 53 19 92 5b 0a 21 e5 32 f7 29 22 8e 03 0c 54 f4 20 87 17 d7 e9 44 c6 cc 76 2e c0 aa 54 05 94 af 9c f1 24 59 ac fb 6b 7c c0 7e 0b b8 65 f8 48 a5 fc 00 46 00 04 00 05 00 2f 00 35 c0 02 c0 04 c0 05 c0 0c c0 0e c0 0f c0 07 c0 09 c0 0a c0 11 c0 13 c0 14 00 33 00 39 00 32 00 38 00 0a c0 03 c0 0d c0 08 c0 12 00 16 00 13 00 09 00 15 00 12 00 03 00 08 00 14 00 11 00 ff 01 00 00 40 00 0b 00 04 03 00 01 02 00 0a 00 34 00 32 00 0e 00 0d 00 19 00 0b 00 0c 00 18 00 09 00 0a 00 16 00 17 00 08 00 06 00 07 00 14 00 15 00 04 00 05 00 12 00 13 00 01 00 02 00 03 00 0f 00 10 00 11 SSL_accept:SSLv3 read client hello A TLS 1.0 Handshake [length 0051], ServerHello 02 00 00 4d 03 01 54 d2 59 b6 3e ad 8a d7 82 e6 ac 2c ed 75 4e 55 c4 ad 68 8a fc 91 45 57 16 33 ed f5 b7 c9 60 0f 20 ea 01 a9 ee 17 71 39 02 70 2c cc 9a 19 af 9b a8 69 4d b4 36 f8 70 0b 17 4f d9 10 e4 46 85 1a 65 00 04 00 00 05 ff 01 00 01 00 write to 0x1a43e90 [0x1a53070] (86 bytes = 86 (0x56)) - 16 03 01 00 51 02 00 00-4d 03 01 54 d2 59 b6 3e Q...M..T.Y. 0010 - ad 8a d7 82 e6 ac 2c ed-75 4e 55 c4 ad 68 8a fc ..,.uNU..h.. 0020 - 91 45 57 16 33 ed f5 b7-c9 60 0f 20 ea 01 a9 ee .EW.3`. 0030 - 17 71 39 02 70 2c cc 9a-19 af 9b a8 69 4d b4 36 .q9.p,..iM.6 0040 - f8 70 0b 17 4f d9 10 e4-46 85 1a 65 00 04 00 00 .p..O...F..e 0050 - 05 ff 01 00 01. 0056 - SPACES/NULS SSL_accept:SSLv3 write server hello A TLS 1.0 Handshake [length 02f3], Certificate 0b 00 02 ef 00 02 ec 00 02 e9 30 82 02 e5 30 82 02 4e 02 09 00 c9 ed cb 4c a7 a1 25 2d 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05 00 30 81 b6 31 0b 30 09 06 03 55 04 06 13 02 55 53 31 0d 30 0b 06 03 55 04 08 13 04 55 74 61 68 31 17 30 15 06 03 55 04 07 13 0e 53 61
Re: NOSRV/BADREQ from some Java based clients
On Sat, Feb 21, 2015 at 12:39 AM, NuSkooler nuskoo...@gmail.com wrote: We have been in the process of deploying HAProxy as a SSL terminator between our client software and back end services. In the testing phases, everything is working great and looking good with one exception: Some old client software that utilizes a Java SSL implementation fail to connect and we end up with logs like this: [20/Feb/2015:15:49:51.632] https_frontend~ https_frontend/NOSRV -1/-1/-1/-1/23 400 187 - - CR-- 0/0/0/0/0 0/0 BADREQ Without HAProxy in the mix, these same clients connect up to our Mochiweb services (via SSL) just fine. Additionally, our newer clients that are OpenSSL based communicate with HAProxy (termination) - Mochiweb (via HTTP) just fine as well. From what I can tell, it appears as though we may have a combination of two bad things: 1) Clients sending some sort of non-standard handshake 3) Mochiweb has been allowing it. Some additional gritty details: * socat 'show errors' shows 0 errors * The same bad clients fail to connect to a OpenSSL s_server (logs below) Since we can't even properly connect to s_server, that may be the end of the road for those clients. However, I'm hoping there may be something that could be configured to allow them through HAProxy. Below is a s_server log. Note the read failure at the end. A similar capture in the view of Wireshark is below that. Lastly, *with* HAProxy when the NOSRV/BADREQ is issued, the client is sent a encrypted 400 Bad Request. Any help/tips appreciated! This represents a large client base that unfortunately cannot be updated for the time being. If we cannot go through HAProxy directly, the next step is to figure out a way to route old clients around it :( Hi, Since HAProxy returns a 400, it means that the issue is above the SSL connection. You should enable HAProxy's stats socket and run the following command on it right after a 400 has been emitted: show errors Then HAProxy will print you why it has blocked the request and why it considered this request was not HTTP compliant. Baptiste
[SPAM] Découvrez le produit idéal pour perdre facilement du poids
Title: PerfectLine 24 Cliquez ici pour lire cet e-mail dans votre navigateur. Dcouvrez LE produit idal pour perdre facilement du poids ! MaigrissezSANS AUCUN EFFORT et surtout SANS DANGER ! - Rducteur d’apptit - Dites stop au grignotage ! - Capteur de graisse et de sucre - Acclrateur de votre transit Se dsinscrire ici
