Increasing amount of spam on the mailing list
Hi All Willy, I am seeing an increasing amount of spam / viral infection data coming across the mailing list. Surely, like surely you don't need an entirely open mailinglist, it's so easy to implement a verification of identity confirmation these days? I am even happy to help in setting this up if you are too busy. J Thanks, Karl. Karl Kloppenborg Head of Development Phone: 1300 884 839 (AU Only - Business Hours) Website: AU http://www.crucial.com.au/ http://www.crucial.com.au | US http://www.crucialp.com http://www.crucialp.com Description: Description: Description: crucial-logo-small image001.gif
Re: nbproc1, ksoftirqd and response time - just can't get it
Hi! First of all thanks too all on this list for helping me to setup and configure haproxy. Thanks to Willy for haproxy. I thought it is a good idea to save somebody like me some time configuring Dell's servers for haproxy to meet DDoS or other traffic-intensive task. Solution is - get rid of bnx2 1Gbit NIC. There is an option in Del''s servers to install Intel's 10Gbit NIC - it is working a way faster (x3, x5) then broadcom. But anyway it is impossible to have one server to filter 10Gbit DDoS attack, at least it is very far from default configuration, so I failed to find solution. But as haproxy is really good software to handle DDoS you can use amazon's cloud. And that was my final solution. Amazon's servers are taking 400Mbit per node, so installing 60 EC2 nodes I've managed to filter DDoS traffic at 24Gbit/s (It was somewhere around 10*10^6 requests per second, or ~150-160k session rate per node) without any problems. Yes, there was some balancing issues, and some failures for couple of minutes, but it's nothing comparing to typical DDoS expirience. And question to Willy: What hardware in your opinion is the best to run haproxy on to serve lot's of HTTP requests 99% percent of which are trash to be blocked? 2011/7/20 Willy Tarreau w...@1wt.eu Hi Dmitriy, On Wed, Jul 20, 2011 at 06:33:27AM +0400, Dmitriy Samsonov wrote: Very strange, but your experience is enourmous so I'd be better to listen to you. As I wrote, before disabling HT I've tested all possible combinatrions of bindings: (haproxy, irq) = (cpu#n, cpu#m) in (0,1), (0,2), (0,3), ..(0,11), (1,0), (1,2), (1,3)...,(1,11) - total of 138 combinations, testing each one for three minutes with two httperf running, and recording information on haproxy's stat socket. If problem was not in HT what then? Given that you tested that many combinations, I would say that there is no option left ! I'm just surprized that you have an hyperthreading which is as bad as the one we had on P4s. At this time, one of the factor for its low efficiency was that the caches were cut in half when it was enabled (half of the cache for each sibling). Maybe this is implemented that way on your CPU, I don't know. (...) Yes, according to /proc/cpuinfo CPU#0 has physical_id = 0, CPU#1 has physical_id = 1, and so on 0, 1, 0, 1... My first idea was to bind haproxy is CPU#0 and process irqs and CPU #2, it should be one CPU in one socket. And looks like these cores shoud share same L2. Yes that's the idea. At least now you know that in case of a DDoS, you can just put your desktop machine in front of your expensive servers to protect them :-) In case of DDoS I should read haproxy's mailing list:) I really appreciate your help. Well, in a few years we managed to stop about a tens of them, maybe even a bit more, including at least three really big ones. Sometimes it can take up to a week to find how to block them. And each time we had to quickly add dirty emergency hacks in the code. Once you've figured how to scale and resist, they often give up because it's an endless game. At high rates, you generally need your internet provider to cooperate (eg: inflate pipes to drain the excess traffic). But that's something pleasant to work on :-) Cheers, Willy
Re: Increasing amount of spam on the mailing list
On 2011-07-26 09:25:42, Karl Kloppenborg wrote: Surely, like surely you don't need an entirely open mailinglist, it's so easy to implement a verification of identity confirmation these days? I filter spam so the main problem I see is bounce messages which are sent to the list for some strange reason. Noted this a few months back. /Allan -- Allan Wind Life Integrity, LLC http://lifeintegrity.com
RE: Increasing amount of spam on the mailing list
Whilst I agree that you can filter, it's not exactly responsible that the mailinglist have this many viral infections running across it... Karl Kloppenborg Head of Development Phone: 1300 884 839 (AU Only - Business Hours) Website: AU http://www.crucial.com.au | US http://www.crucialp.com -Original Message- From: Allan Wind [mailto:allan_w...@lifeintegrity.com] Sent: Tuesday, 26 July 2011 10:34 AM To: haproxy@formilux.org Subject: Re: Increasing amount of spam on the mailing list On 2011-07-26 09:25:42, Karl Kloppenborg wrote: Surely, like surely you don't need an entirely open mailinglist, it's so easy to implement a verification of identity confirmation these days? I filter spam so the main problem I see is bounce messages which are sent to the list for some strange reason. Noted this a few months back. /Allan -- Allan Wind Life Integrity, LLC http://lifeintegrity.com
Re: nbproc1, ksoftirqd and response time - just can't get it
Hi Dmitriy, On Tue, Jul 26, 2011 at 03:51:20AM +0400, Dmitriy Samsonov wrote: Hi! First of all thanks too all on this list for helping me to setup and configure haproxy. Thanks to Willy for haproxy. I thought it is a good idea to save somebody like me some time configuring Dell's servers for haproxy to meet DDoS or other traffic-intensive task. Solution is - get rid of bnx2 1Gbit NIC. Hehe :-) There is an option in Del''s servers to install Intel's 10Gbit NIC - it is working a way faster (x3, x5) then broadcom. Intel's 10G NICs are fast, but generally hard to tune, you have a hard trade-off between data rate and latency. Basically you tune them for large or small packets but not both. Still for DDoS they might be well suited. But anyway it is impossible to have one server to filter 10Gbit DDoS attack, at least it is very far from default configuration, so I failed to find solution. 10G DDoS is something very hard to resist ! But as haproxy is really good software to handle DDoS you can use amazon's cloud. And that was my final solution. Amazon's servers are taking 400Mbit per node, so installing 60 EC2 nodes I've managed to filter DDoS traffic at 24Gbit/s (It was somewhere around 10*10^6 requests per second, or ~150-160k session rate per node) without any problems. Yes, there was some balancing issues, and some failures for couple of minutes, but it's nothing comparing to typical DDoS expirience. Wow! I'm impressed! But did you perform the tests from within Amazon's cloud or from the outside ? I don't know what their external peering is, and I'm wondering how you managed to send 24 Gbps over the internet. Also, what's the cost of running 60 of these nodes ? And question to Willy: What hardware in your opinion is the best to run haproxy on to serve lot's of HTTP requests 99% percent of which are trash to be blocked? My best results were achieved on Core i5 3.3 GHz (dual core, 4 threads). In short, I bind network IRQs to core zero and haproxy to core one and the two remaining threads ensure there is some rope left for system management, SSH, etc. With this you can run at 100% CPU all the day if you want. I managed to get haproxy to dynamically filter 300k connections per second on such a configuration. That's SYN, SYN/ACK, ACK, RST, based on a source IP's connection rate. Depending on the type of DDoS you're dealing with, it may be worth distributing the NIC's IRQs to multiple cores so that the kernel's work can scale (eg: emit a SYN/ACK). It might even be worth having multiple haproxy processes, because when you're blocking a DDoS, you don't really mind about monitoring, stats, health checks, etc... You only want your machine to be as fast as possible, and doing so is possible with multi-queues. You then need to spread your NICs IRQs to all cores and bind as many haproxy processes as cores (and maually pin them). The CPU-NIC affinity will not be good on the backend but that's not the issue since you're dealing with a frontend where you want to reject most of your traffic. Best regards, Willy
Re: Increasing amount of spam on the mailing list
I love the suggestion and offer to administrate the mail list (and I too volunteer), but, ultimately: whatever. SPAM is part of most any list and the more time the guys spend on one of the best pieces of software in the world, the better. I happily skip these messages in hopes Willy Cyril and the guys never care about wasting their time with mailman and postgix plugins or whatever this list uses. On Monday, July 25, 2011, Karl Kloppenborg k...@crucialp.com wrote: Whilst I agree that you can filter, it's not exactly responsible that the mailinglist have this many viral infections running across it... Karl Kloppenborg Head of Development Phone: 1300 884 839 (AU Only - Business Hours) Website: AU http://www.crucial.com.au | US http://www.crucialp.com -Original Message- From: Allan Wind [mailto:allan_w...@lifeintegrity.com] Sent: Tuesday, 26 July 2011 10:34 AM To: haproxy@formilux.org Subject: Re: Increasing amount of spam on the mailing list On 2011-07-26 09:25:42, Karl Kloppenborg wrote: Surely, like surely you don't need an entirely open mailinglist, it's so easy to implement a verification of identity confirmation these days? I filter spam so the main problem I see is bounce messages which are sent to the list for some strange reason. Noted this a few months back. /Allan -- Allan Wind Life Integrity, LLC http://lifeintegrity.com
