Re: Wrong sha256 checksum for HAProxy 1.8 and 1.9?
Hi Kevin, On Tue, Feb 26, 2019 at 06:27:30AM +, Kevin Mao wrote: > Hi haproxy@, > > It seems like the sha256 checksum's are wrong for the latest 1.8 and 1.9 > HAProxy versions, Can you please confirm? > https://www.haproxy.org/download/1.9/src/, > $ shasum -a 256 -c haproxy-1.9.4.tar.gz.sha256haproxy-1.9.4.tar.gz: > FAILEDshasum: WARNING: 1 computed checksum did NOT match > $ shasum -a 256 -c haproxy-1.9.3.tar.gz.sha256haproxy-1.9.3.tar.gz: > FAILEDshasum: WARNING: 1 computed checksum did NOT match > $ shasum -a 256 -c haproxy-1.8.19.tar.gz.sha256haproxy-1.8.19.tar.gz: > FAILEDshasum: WARNING: 1 computed checksum did NOT match > $ shasum -a 256 -c haproxy-1.8.18.tar.gz.sha256haproxy-1.8.18.tar.gz: > FAILEDshasum: WARNING: 1 computed checksum did NOT match It's not what I'm seeing here using the sha256sum utility : $ cat haproxy-1.9.4.tar.gz.sha256 8483fe12b30256f83d542b3f699e165d8f71bf2dfac8b16bb53716abce4ba74f haproxy-1.9.4.tar.gz $ sha256sum haproxy-1.9.4.tar.gz 8483fe12b30256f83d542b3f699e165d8f71bf2dfac8b16bb53716abce4ba74f haproxy-1.9.4.tar.gz $ sha256sum -c haproxy-1.9.4.tar.gz.sha256 haproxy-1.9.4.tar.gz: OK OpenSSL agrees with it : $ openssl sha256 haproxy-1.9.4.tar.gz SHA256(haproxy-1.9.4.tar.gz)= 8483fe12b30256f83d542b3f699e165d8f71bf2dfac8b16bb53716abce4ba74f So I suspect that you're facing a download issue or your shasum fails for whatever reason. The file above is 2357935 bytes long here. Please check that matches on your side. thanks, Willy
Wrong sha256 checksum for HAProxy 1.8 and 1.9?
Hi haproxy@, It seems like the sha256 checksum's are wrong for the latest 1.8 and 1.9 HAProxy versions, Can you please confirm? https://www.haproxy.org/download/1.9/src/, $ shasum -a 256 -c haproxy-1.9.4.tar.gz.sha256haproxy-1.9.4.tar.gz: FAILEDshasum: WARNING: 1 computed checksum did NOT match $ shasum -a 256 -c haproxy-1.9.3.tar.gz.sha256haproxy-1.9.3.tar.gz: FAILEDshasum: WARNING: 1 computed checksum did NOT match $ shasum -a 256 -c haproxy-1.8.19.tar.gz.sha256haproxy-1.8.19.tar.gz: FAILEDshasum: WARNING: 1 computed checksum did NOT match $ shasum -a 256 -c haproxy-1.8.18.tar.gz.sha256haproxy-1.8.18.tar.gz: FAILEDshasum: WARNING: 1 computed checksum did NOT match Thanks,Kevin
Is `ssl_fc` set when using `send-proxy-v2-ssl` and the original connection was over TLS? (HAProxy<->HAProxy setup)
Hello all! As the subject anticipates, I have (on HAProxy 1.8.14) the following setup: * an "upstream" HAProxy instance listening both on HTTP and HTTPS, which sends to a beckend that has configured `send-proxy-v2-ssl`; * a "downstream" HAProxy instance listening with `accept-proxy`; Apparently the `ssl_fc` on the downstream HAProxy is always set to false even though the original request originates over TLS, and the PROXY v2 protocol seems to correctly contain this information as seen from the network capture bellow. 0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a 21 11 00 23 ...Q UIT.!..# 0010 4f 76 0a b9 55 5a f4 f9 d9 ea 01 bb 01 00 02 68 Ov..UZ.. ...h 0020 32 20 00 0f 01 00 00 00 00 21 00 07 54 4c 53 76 2 .. .!..TLSv 0030 31 2e 32 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 1.2GET / HTTP/1. 0040 31 0d 0a 75 73 65 72 2d 61 67 65 6e 74 3a 20 63 1..user- agent: c 0050 75 72 6c 2f 37 2e 36 33 2e 30 0d 0a 61 63 63 65 url/7.63 .0 Thus my question is if `ssl_fc` is set to true only if the "current" transport is actually over TLS? And my second question, is there an ACL that evaluates to true if the PROXY v2 protocol was used, and the TLS is present in the meta-data. (I can obviously use the frontend port, which is correctly configured to 443, to bypass `ssl_fc`, however I was looking for a more "definitive" solution.) Thanks, Ciprian.
Re: High p99 latency with HAProxy 1.9 in http mode compared to 1.8
Any ideas on this ? Seeing issues with HAProxy 1.9 performance with connection pooling turned on. Thanks. On Wed, Feb 13, 2019 at 3:46 PM Ashwin Neerabail wrote: > Hi Willy, > > Thank you for the detailed response. Sorry for the delay in response. > > I ran all the combinations multiple times to ensure consistent > reproducibility. > Here is what i found : > > Test Setup (same as last time): > 2 Kube pods one running Haproxy 1.8.17 and another running > 1.9.2 loadbalancing across 2 backend pods. > Haproxy container is given 1 CPU , 1 GB Memory. > 500 rps per pod test , latencies calculated for 1 min window. > > - previous results for comparison > * Haproxy 1.9 - p99 is ~ 20ms , p95 is ~ 11ms , median is 5.5ms > * Haproxy 1.8 - p99 is ~ 8ms , p95 is ~ 6ms, median is 4.5ms > * Haproxy 1.9 : Memory usage - 130MB , CPU : 55% util > * Haproxy 1.8 : Memory usage - 90MB , CPU : 45% util > > - without SSL > HAProxy 1.8 performs slightly better than 1.9 > * Haproxy 1.9 - p99 is ~ 9ms , p95 is ~ 3.5ms , median is 2.3ms* > * Haproxy 1.8 - p99 is ~ 5ms , p95 is ~ 2.5ms, median is 1.7ms* > CPU Usage is identical. (0.1% CPU) > > - by disabling server-side idle connections (using "pool-max-conn 0" on > the server) though "http-reuse never" should be equivalent > > This seems to have done the trick. Adding `pool-max-conn 0` or `http-reuse > never` fixes the problem. > 1.8 and 1.9 perform similarly (client app that calls haproxy is using > connection pooling). *Unfortunately , we have legacy clients that close > connections to front end for every request.* > CPU Usage for 1.8 and 1.9 was same ~22%. > >- by placing an inconditional redirect rule in your backend so that we > check how it performs when the connection doesn't leave : > http-request redirect location / > > Tried adding monitor-uri and returning from remote haproxy rather than > hitting backend server. > Strangely , in this case I see nearly identical performance /CPU usage > with 1.8 and 1.9 even with http reuse set to aggressive. > CPU Usage for 1.8 and 1.9 was same ~35%. > *Set up is Client > HAProxy > HAProxy (with monitor-uri) > Server.* > > If you're running harmless tests, you can pick the latest nightly snapshot > of 2.0-dev which is very close to what 1.9.4 will be. > I also tried the perf tests with 2.0-dev. It shows the same behavior as > 1.9. > > If you have potential fixes / settings / other debugging steps that can be > tweaked - I can test them out and publish the results. > Thanks for your help. > > -Ashwin > > > On Thu, Jan 31, 2019 at 1:43 PM Willy Tarreau wrote: > >> Hi Ashwin, >> >> On Thu, Jan 31, 2019 at 10:32:33AM -0800, Ashwin Neerabail wrote: >> > Hi, >> > >> > We are in process of upgrading to HAProxy 1.9 and we are seeing >> consistent >> > high latency with HAProxy 1.9.2 as compared to 1.8.17 when using HTTP >> Mode >> > ( both with and without TLS). However no latency issues with TCP Mode. >> > >> > Test Setup: >> > 2 Kube pods one running Haproxy 1.8.17 and another running 1.9.2 >> > loadbalancing across 2 backend pods. >> > Haproxy container is given 1 CPU , 1 GB Memory. >> > 500 rps per pod test , latencies calculated for 1 min window. >> > >> > Latencies as measured by client: >> > >> > *When running TCP Mode , the p99 latency between 1.9 and 1.8 is the >> same.* >> > *When running HTTP Mode (with TLS),* >> > *Haproxy 1.9 - p99 is ~ 20ms , p95 is ~ 11ms , median is 5.5ms* >> > *Haproxy 1.8 - p99 is ~ 8ms , p95 is ~ 6ms, median is 4.5ms* >> >> The difference is huge, I'm wondering if it could be caused by a last TCP >> segment being sent 40ms too late once in a while. Otherwise I'm having a >> hard time imagining what can take so long a time at 500 Rps! >> >> In case you can vary some test parameters to try to narrow this down, it >> would be interesting to try again : >>- without SSL >>- by disabling server-side idle connections (using "pool-max-conn 0" on >> the server) though "http-reuse never" should be equivalent >>- by placing an inconditional redirect rule in your backend so that we >> check how it performs when the connection doesn't leave : >> http-request redirect location / >> >> > This increased latency is reproducible across multiple runs with 100% >> > consistency. >> > Haproxy reported metrics for connections and requests are the same for >> both >> > 1.8 and 1.9. >> > >> > Haproxy 1.9 : Memory usage - 130MB , CPU : 55% util >> > Haproxy 1.8 : Memory usage - 90MB , CPU : 45% util >> >> That's quite interesting, it could indicate some excessive SSL >> renegotiations. Regarding the extra RAM, I have no idea though. It could >> be the result of a leak though. >> >> Trying 1.9.3 would obviously help, since it fixes a number of isses, even >> if at first glance I'm not spotting one which could explain this. And I'd >> be interested in another attempt once 1.9.4 is ready since it fixes many >> backend-side connection issues. If you're running harmless tests,
Re: [Potential Spoof] [PATCH] BUG/MAJOR: fd/threads, task/threads: ensure all spin locks are unlocked
Hi Richard, On Wed, Feb 20, 2019 at 11:58:42PM +, Richard Russo wrote: > While continuing to test this, I ended up with a crash in > listener.c:listener_accept on a closed/closing listen socket where > fdtab[fd].owner is NULL by the time the thread gets there. This is possible > because the fd.c: fdlist_process_cached_events unlocks the spinlock before > calling fdtab[fd].iocb(fd); allowing for a concurrent close. > > I'm not sure if the right thing to do is to hold the FD's spinlock while > calling the callback, consider a read/write style lock for listen FDs, or > simply adding a check that l is not NULL in listener_accept similar to the > one in connection.c:conn_fd_handle. I'm testing the NULL check right now, > since it's simplest. I haven't had a crash since, but that doesn't mean it > won't crash. This change would need to be applied to 1.8 as well, although I > haven't managed to observe a crash on 1.8. I think you're right on all accounts. I've pushed your patch, and after talking with Willy, came to the conclusion checking if the listener is NULL before using it should be enough, so I pushed a patch that did just that as well. Can you confirm that your problems are gone in master ? Thanks a lot ! Olivier
