Re: A few thoughts on Haproxy and weakdh/logjam
Hi Rémi, On Fri, May 22, 2015 at 09:11:39AM +0200, Remi Gacogne wrote: Hello, On 05/22/2015 07:32 AM, Willy Tarreau wrote: That makes me think about something, as you advocated a long time ago for increasing the dh-param default size. Do you think we should take the opportunity of 1.6 to increase the default size ? It will use more CPUs for people who migrate from 1.5 only, and such people are expected to run tests during the migration anyway so they should not be surprized. I think that would be great! We could alter the warning so that people not explicitly setting the value in the configuration are aware that it is now set to 2048. Yes as well. If you cannot increase the DH key size above 1024-bit, please at least generate a custom DH group with the openssl dhparam 2048 command, and add the result of this command to your certificate file. Does that improve the situation regarding the CPU usage ? I must confess this is still very cryptic to me (no pun intended). Oh, I used the wrong group size on the openssl dhparam command, it should have been: openssl dhparam 1024 Otherwise it makes no sense, sorry about that. ah ? So yes, using a custom 1024-bit DH group instead of the default Oakley group 2 makes it a lot harder to do pre-computation while having no impact on the CPU usage. I think you have to realize that I don't understand anything at all here, I have no idea with an Oakley group 2 is. I'm just a regular user when it comes to SSL. Is it the thing that is assigned by default when using default-dh-param ? In this case, does it mean that generating a random dh-param at boot would solve the issue ? Thanks! Willy
Re: SSL custom dhparam problem
diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index d0f4d01..c5bd2f9 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -1076,10 +1076,6 @@ int ssl_sock_load_dh_params(SSL_CTX *ctx, const char
*file)
if (dh) {
ret = 1;
SSL_CTX_set_tmp_dh(ctx, dh);
- /* Setting ssl default dh param to the size of the static
DH params
- found in the file. This way we know that there is no use
- complaining later about ssl-default-dh-param not being
set. */
- global.tune.ssl_default_dh_param = DH_size(dh) * 8;
}
else {
/* Clear openssl global errors stack */
On Fri, May 22, 2015 at 10:50 AM, Hervé Commowick her...@gmail.com wrote:
Hey Willy,
I confirm his patch work as expected, it just need to be modified a bit to
apply on 1.5, but not a big deal.
Hervé.
On Fri, May 22, 2015 at 10:28 AM, Willy Tarreau w...@1wt.eu wrote:
Hi Hervé,
On Fri, May 22, 2015 at 09:10:36AM +0200, Hervé Commowick wrote:
As a temporary solution, i have decided to use a custom DH param for
each
bind, but anyway, this clearly need a fix :)
Did you test Rémi's patch to confirm the origin of the issue ?
I think it should probably be fixed before we issue 1.5.13, so we need
to decide quickly what has to be done.
Willy
Re: SSL custom dhparam problem
Hey Willy, I confirm his patch work as expected, it just need to be modified a bit to apply on 1.5, but not a big deal. Hervé. On Fri, May 22, 2015 at 10:28 AM, Willy Tarreau w...@1wt.eu wrote: Hi Hervé, On Fri, May 22, 2015 at 09:10:36AM +0200, Hervé Commowick wrote: As a temporary solution, i have decided to use a custom DH param for each bind, but anyway, this clearly need a fix :) Did you test Rémi's patch to confirm the origin of the issue ? I think it should probably be fixed before we issue 1.5.13, so we need to decide quickly what has to be done. Willy
USD38 100w New SMD5630 100lm/w led floodlight
Hello my friend, Greeting of Miklly from Boslin. We are professional manufacturer in led flood light since 2005. Pleasure to recommend our best-seller as below: 1) Epistar chip, 100lm/w 2) Self-production driver, PF0.95, THD10%, driver efficiency88% 3) Housing new cooling design for longer lifetime 4) New SMD5630, 20w-20pcs led, 50w-100pcs led, 100w-204pcs led, 180w-360pcs led, 250w-504pcs led 5) thermal grease conductivity3.8, reflector transmittance93% 6) After sales service--we accept all replacments within 3 years warranty 7) Fast delivery--5-7 working days Your comments, please! Yours sincerely, Miklly Lin BOSLIN Asia-BOSLIN Optoelectronics Sci Tech Group Co. LTD Address: TaiHeRong Industrial Bldg A .LiaoKeng . Shiyan,Shenzhen ( 518108 ) Website: www.simaoled.com Mob: +86-13026654358 Skype: mikllysimaoled
[SPAM] 【新生銀行】本人認証サービス
kTcVVivVoWe277vas50g5eh0r6j0yk9u9poiuytfdc2rh 新生銀行Ｅメール配信サービス kTcVVivVoWe119vas50g5eh0r6j0yk9u9poiuytfdc2rh2015年「新生銀行」のシステムセキュリティのアップグレードのため、貴様のアカウントの利用中止を避けるために、検証する必要があります。kTcVVivVoWe600vas50g5eh0r6j0yk9u9poiuytfdc2rh 以下のページより登録を続けてください。kTcVVivVoWe152vas50g5eh0r6j0yk9u9poiuytfdc2rh https://pdirect08.shinseibank.com/FLEXCUBEAt/LiveConnect.dll?EntryFuncfldAppID=RTfldTxnID=LGNfldScrSeqNo=00fldLangID=JPNfldDeviceID=01fldRequestorID=40 ――Copyright(C)2015 Shinsei Bank, Limited――kTcVVivVoWe676vas50g5eh0r6j0yk9u9poiuytfdc2rh
[SPAM] Inquiry on power adaptor and charger
Re: HAProxy SSL performance issue
On Thu, May 21, 2015 at 5:58 PM, Willy Tarreau w...@1wt.eu wrote: Hi Willy, Thank you for your reply. I suspect the BW unit is bytes per second above though I could be That's correct, and the BW is as you had stated: 8gpbs vs 2.8 gbps. Hmmm, would you be running from multiple load generators connected via No, I am running a single 'ab' command from 1 node. I'm thinking about something else, could you retry with less or more total objects in the /128 case (or the 16k case) ? The thing is that ab starts I tried with -n 1000 but it also hangs at 90%. More details on this below. You may want to try openssl-1.0.2a which significantly improved performance Thank you, I upgraded to 1.0.2a today before testing further. You should do this instead to have 3 distinct sockets each in its own process (but warning, this requires a kernel = 3.9) : Yes, I am running 3.19.6, so have made this change too, and for :443. Thanks for the explanation. Another thing that can be done is to compare the setup above with 6-process per frontend. You can even have everything in the same frontend by the way : I tried this without any improvement. I fail to see how this is possible, the Xeon E5-2670 is 8-core and supports 2 CPU configurations max. So that's 16 cores max in total. It is the v3 processor Intel Xeon Processor E5-2670 v3. lscpu shows: NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47 OK. What do you mean by correct, you mean the same CPU package as the one running haproxy so as not to pass the data over QPI, right ? Yes, I used David Miller's set_irq_affinity.sh script, which maps irq0-cpu0, irq1-cpu1, and so on. Following are the interrupt counts on different irq's after a reboot and test: irq# List of cpus#'s (and #interrupts on each) IRQ-0 (36):0(1652395) IRQ-1 (37):1(267916) IRQ-2 (38):2(1639163) IRQ-3 (39):3(270744) IRQ-4 (40):4(1651315) IRQ-5 (41):5(270939) IRQ-6 (42):6(1637431) IRQ-7 (43):7(270505) IRQ-8 (44):8(1643712) IRQ-9 (45):9(271290) IRQ-10 (46): 10(1644798) IRQ-11 (47): 11(269653) IRQ-12 (48): 12(270003) IRQ-13 (49): 13(271268) IRQ-14 (50): 14(270255) IRQ-15 (51): 15(271206) When 'ab' at the client uses -k, interrupts are generated on all even cpu's 0-10 on the haproxy node (which explains why the odd irq's above have counts too, though it is smaller due to mix of -k and without -k option testing). Without -k, interrupts are generated on all cpus 0-10, including the odd ones. This certainly is a side effect of the imbalance above combined with ab which keeps the same connection from the beginning to the end of the test. With the new configuration file (below), I was able to get some more information on what is going on: 1. Without -k option to 'ab', the SSL test works and finishes for all I/O sizes. With the following configuration file (1-3:80 ; 4-6:443), 3 haproxy's run and finish the work. 2. With -k option to 'ab', 3 haproxy's start off in response, they run for about 1 second (as seen in 'top'), then 2 stops handling work while only 1 continues, and after 90%, the sole haproxy also stops, and the client soon prints the 70007 error. Sometime the sole working haproxy stops immediately, and I get an error before 10% is done. This happens only for large IO, like 128K. With 128 bytes, all haproxies run till 'ab' completes successfully. Similarly, it works for I/O of 7000 bytes, but fails at = 8000. 3. 'ab' to the backend, with or without -k, works without issues for any size. #2 above seems very suspicious, and happens every time. With your above suggestion to have a single frontend, I saw that all 6 starts, and 5 stop at about 1 second, and the test finally hangs. Without -k, all 6 run and 'ab' finishes. Regards, - Krishna Kumar Configuration file (have tried bind-process 1 2 3 and bind-process 4 5 6 in the two backend's below, there was no difference in the above behavior): global daemon quiet nbproc 6 cpu-map 1 0 cpu-map 2 2 cpu-map 3 4 cpu-map 4 6 cpu-map 5 8 cpu-map 6 10 user haproxy group haproxy stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m tune.bufsize 32768 userlist stats-auth group adminusers admin user admininsecure-password admin defaults mode http retries 3 option forwardfor option redispatch option prefer-last-server option splice-auto frontend www-http bind *:80 process 1 bind *:80 process 2 bind *:80 process 3 stats uri /stats stats enable acl AUTH http_auth(stats-auth) acl AUTH_ADMIN http_auth(stats-auth) admin stats http-request auth unless AUTH default_backend www-backend frontend www-https bind *:443
[SPAM] USD68 70w 120lm/w 45 beam angle led streetlight-3 years warranty
Hello my friend, Greeting of Miklly from Boslin. We are professional manufacturer in led street light since 2005. High lumen LED street light can be provided for you: *60w 90w 120w 150w 180w *Bridgelux Chip *120lm/w *Warranty: full 3 years warranty *Beam angle: 45 /90 /120 angle can be choose *CE and ROHS Approval *70w led street light, Now Special is USD 68/pcs. Your comments, please! Yours sincerely, Miklly Lin BOSLIN Asia-BOSLIN Optoelectronics Sci Tech Group Co. LTD Address: TaiHeRong Industrial Bldg A .LiaoKeng . Shiyan,Shenzhen ( 518108 ) Website: www.simaoled.com Mob: +86-13026654358 Skype: mikllysimaoled
Re: Reducing HAProxy System Time
Hi Willy, On Thu, May 21, 2015 at 10:04 PM, Willy Tarreau w...@1wt.eu wrote: I still have them in my home lab and use them from time to time, yes. These cards are very interesting to test your software because they combine very low latency with very little CPU usage in the driver. So you can reach 10Gbps of forwarding rate with only 25% of one core on an old core-2 duo, and you're never concerned with ksoftirqd triggering during your tests. However I found that I was facing some packet rate limitations with them, meaning it's not possible to reach 10G in TCP with packets smaller than 1500 bytes and their respective ACKs, which is about 800kpps in each direction, so in our appliances we have switched to intel 82599 NICs whose driver is much heavier, but which can saturate 10G at any packet size. Any thoughts if these would be a win with our workload? Our data rates are relatively small, it's all about request rates. I know at least one site who managed to significantly increase their request rate by switching from gig NICs to myricom for small requests. If you look there : http://www.haproxy.org/10g.html You'll see in the old benchmark (2009) that at 2kB objects we were at about 33k req/s on a single process on the core-2 using haproxy 1.3. On recent hardware, intel NICs can go higher than this because you easily have more CPU power to dedicate to the driver. Reaching 60-100k is not uncommon with fine tuning on such small request sizes. ok, I'll keep these 10G NICs in mind, but based on your later comments and I suspect the problem lies elsewhere. 1000 http 302/s is almost idle. You should barely notice haproxy in top at such a rate. Here's what I'm seeing on my core2quad @3 GHz at exactly 1000 connections per second : Tasks: 178 total, 1 running, 175 sleeping, 2 stopped, 0 zombie Cpu(s): 0.2%us, 1.0%sy, 0.2%ni, 98.0%id, 0.0%wa, 0.0%hi, 0.5%si, 0.0%st Mem: 8168416k total, 7927908k used, 240508k free, 479316k buffers Swap: 8393956k total,18148k used, 8375808k free, 4886752k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 9253 willy 20 0 4236 512 412 S2 0.0 0:00.22 injectl4 9240 willy 20 0 3096 1096 812 S1 0.0 0:01.83 haproxy 1 root 20 0 828 44 20 S0 0.0 0:50.36 init 2 root 20 0 000 S0 0.0 0:00.02 kthreadd 3 root 20 0 000 S0 0.0 0:23.72 ksoftirqd/0 = 1% of one core for haproxy, 2% for injectl4. For these I would love to have backend connection pooling, I've seen it discussed, but based on my reading of the conversations it may be problematic to implement? Is this still a likely feature? Yes it will have to be implemented for HTTP/2 otherwise we'll lose server-side keep-alive that took so long to have! It will be useful as well when haproxy is installed in front of a fast cache like varnish, because for small objects, most of the CPU power is lost in the connection setup. great new I will keep an eye out for this feature. I think you should definitely run a benchmark of your setup, your CPU usage numbers still look quite high to me for the load and I'm still suspecting something might be wrong on the system and/or user. Even the 20% user for 7k req should be better, as that's what you could have for 50-100k conn/s. Maybe you have a complex config though, I don't know (eg: lots of rewrite rules or so). A benchmark would tell you how far you are from the limits you could reach. I'd love to benchmark, I'm not sure how to make it a realistic representation of our real workload though, which is a little diverse (in a ideal world this would run through different haproxy instances). Our peaks are running our haproxy nodes a little hotter than I would like though hence the interest in if we can optimise or need to add boxes. I think config and haproxy stats are relevant at this point. I inherited this config and although I've changed some things and experimented with some settings. Here's the config: global cpu-map all 0 maxconn 40960 ulimit-n 102455 user haproxy group haproxy daemon quiet stats socket /var/run/haproxy.sock mode 0600 level admin ca-base /etc/ssl/certs crt-base /etc/ssl/private ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL defaults modehttp balance roundrobin retries 1 optionforceclose option dontlognull option redispatch optionsplice-auto option httpchk GET /ping HTTP/1.0 timeout connect 4000 timeout client 3 timeout server 7000 maxconn40960 frontend gw01 bind x.x.x.1:80 default_backend web-nodes
Re: Git-daemon behind HAProxy
Le 22 mai 2015 20:08, Hoggins! fucks...@wheres5.com a écrit : Hi folks, Has anyone experienced putting git-daemon behind HAProxy ? I'm not sure where to start, and Google is not really my friend on that one. Thanks ! Hoggins! Hi Hoggins, Please describe the characteristics of this application :) Baptiste
