RE: [WISPA] SSH DOS Killing Linux
The infected sub was bandwidth managed with HTB to 256k cir, 1 mbps mir, but not anything for PPS. Tom- Why don't you just limit the number PPS at the customers radio? Marty -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tom DeReggi Sent: Saturday, January 06, 2007 9:27 PM To: WISPA General List Subject: [WISPA] SSH DOS Killing Linux We recently had a really nasty DOS attack that took down a large part of our network across several cell sites, from the infected client all the way to the Internet transit. Take note that we identified the problem quickly and cured it quickly. But This is the first time that this has occured in 5 years, as we have a good number of smart design characteristics that have limited the effects of most viruses on our network. We stopped the attack, by blocking SSH to the infected sub. The average amount of traffic crossing the entire network path from the client to the Internet was about 500 kbps on average. (This was a 20 mbps wireless link, and a 100mbps fiber trnasport link to the transit.). The two routers were a P4 2Ghz, and a Dual XEON 2.2Ghz w/ 10,000rpm SCSI3. The damage was that the CPU was nailed on both routers to about 99.9% using TOP to monitor stats. We varified that successful SSH sessions were not made directly to the protected routers themselves. Take note that the wireless links were barely effected, it was the router 2 hops away (Dual XEON) that got over loaded the most. Our routers have been tested to pass over 2 gbps of throughput easilly. And have been load tested to survive very small packets and high PPS adequately. The infected sub was bandwidth managed with HTB to 256k cir, 1 mbps mir, but not anything for PPS. So I'm looking for reasons that the CPU got overloaded. My theory is that the DOS attack resulted in a large number of disk writes, ( maybe logging?) causing the CPU saturation. I've had a hard time locating the cause. And have not discovered which virus yet, although I should have more info soon from my clients. So my question What needs to be done on a Linux machine to harden it, to protect against CPU oversaturation, during DOS attacks? What should and shouldn't be logged? Connection Tracking? Firewall logging? Traffic stats? Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] SSH DOS Killing Linux
And how would I do that? Yes I know, I think that is a VL feature, and my radio is not VL. If I were able to limit the PPS then that would solve the problem. But technically why should I have to limit the PPS, because the radios themselves are no where near getting saturated by the amount of PPS currently going through. What is getting saturated is the HDD based XEON rotuers. My point here is that a XEON base GB router should not be able to handle less PPS than a 100Mhz Pentium based Radio. I should be able to tweak our Linux configuration to solve the problem and allow the Linux box to run optimally without risk. Lastly, what is the appropriate PPS limit that would not compromise a custoemr's traffic? Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Marty Dougherty [EMAIL PROTECTED] To: 'WISPA General List' wireless@wispa.org Sent: Sunday, January 07, 2007 8:24 AM Subject: RE: [WISPA] SSH DOS Killing Linux The infected sub was bandwidth managed with HTB to 256k cir, 1 mbps mir, but not anything for PPS. Tom- Why don't you just limit the number PPS at the customers radio? Marty -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tom DeReggi Sent: Saturday, January 06, 2007 9:27 PM To: WISPA General List Subject: [WISPA] SSH DOS Killing Linux We recently had a really nasty DOS attack that took down a large part of our network across several cell sites, from the infected client all the way to the Internet transit. Take note that we identified the problem quickly and cured it quickly. But This is the first time that this has occured in 5 years, as we have a good number of smart design characteristics that have limited the effects of most viruses on our network. We stopped the attack, by blocking SSH to the infected sub. The average amount of traffic crossing the entire network path from the client to the Internet was about 500 kbps on average. (This was a 20 mbps wireless link, and a 100mbps fiber trnasport link to the transit.). The two routers were a P4 2Ghz, and a Dual XEON 2.2Ghz w/ 10,000rpm SCSI3. The damage was that the CPU was nailed on both routers to about 99.9% using TOP to monitor stats. We varified that successful SSH sessions were not made directly to the protected routers themselves. Take note that the wireless links were barely effected, it was the router 2 hops away (Dual XEON) that got over loaded the most. Our routers have been tested to pass over 2 gbps of throughput easilly. And have been load tested to survive very small packets and high PPS adequately. The infected sub was bandwidth managed with HTB to 256k cir, 1 mbps mir, but not anything for PPS. So I'm looking for reasons that the CPU got overloaded. My theory is that the DOS attack resulted in a large number of disk writes, ( maybe logging?) causing the CPU saturation. I've had a hard time locating the cause. And have not discovered which virus yet, although I should have more info soon from my clients. So my question What needs to be done on a Linux machine to harden it, to protect against CPU oversaturation, during DOS attacks? What should and shouldn't be logged? Connection Tracking? Firewall logging? Traffic stats? Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] SSH DOS Killing Linux
Thanks Steve! I think that should help alot. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Steve [EMAIL PROTECTED] To: WISPA General List wireless@wispa.org Sent: Sunday, January 07, 2007 11:52 AM Subject: Re: [WISPA] SSH DOS Killing Linux Have you installed software such as fail2ban which will block the ip address after n number of failed ssh logins for n number of seconds. Depending on the purpose of the server it may block internet access for the client, but I wouldn't worry about that for my network. I have it installed on all my linux boxes and it blocks the routine ssh attacks that are all too common these days. -- Tom DeReggi wrote: We recently had a really nasty DOS attack that took down a large part of our network across several cell sites, from the infected client all the way to the Internet transit. Take note that we identified the problem quickly and cured it quickly. But This is the first time that this has occured in 5 years, as we have a good number of smart design characteristics that have limited the effects of most viruses on our network. We stopped the attack, by blocking SSH to the infected sub. The average amount of traffic crossing the entire network path from the client to the Internet was about 500 kbps on average. (This was a 20 mbps wireless link, and a 100mbps fiber trnasport link to the transit.). The two routers were a P4 2Ghz, and a Dual XEON 2.2Ghz w/ 10,000rpm SCSI3. The damage was that the CPU was nailed on both routers to about 99.9% using TOP to monitor stats. We varified that successful SSH sessions were not made directly to the protected routers themselves. Take note that the wireless links were barely effected, it was the router 2 hops away (Dual XEON) that got over loaded the most. Our routers have been tested to pass over 2 gbps of throughput easilly. And have been load tested to survive very small packets and high PPS adequately. The infected sub was bandwidth managed with HTB to 256k cir, 1 mbps mir, but not anything for PPS. So I'm looking for reasons that the CPU got overloaded. My theory is that the DOS attack resulted in a large number of disk writes, ( maybe logging?) causing the CPU saturation. I've had a hard time locating the cause. And have not discovered which virus yet, although I should have more info soon from my clients. So my question What needs to be done on a Linux machine to harden it, to protect against CPU oversaturation, during DOS attacks? What should and shouldn't be logged? Connection Tracking? Firewall logging? Traffic stats? Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
[WISPA] SSH DOS Killing Linux
We recently had a really nasty DOS attack that took down a large part of our network across several cell sites, from the infected client all the way to the Internet transit. Take note that we identified the problem quickly and cured it quickly. But This is the first time that this has occured in 5 years, as we have a good number of smart design characteristics that have limited the effects of most viruses on our network. We stopped the attack, by blocking SSH to the infected sub. The average amount of traffic crossing the entire network path from the client to the Internet was about 500 kbps on average. (This was a 20 mbps wireless link, and a 100mbps fiber trnasport link to the transit.). The two routers were a P4 2Ghz, and a Dual XEON 2.2Ghz w/ 10,000rpm SCSI3. The damage was that the CPU was nailed on both routers to about 99.9% using TOP to monitor stats. We varified that successful SSH sessions were not made directly to the protected routers themselves. Take note that the wireless links were barely effected, it was the router 2 hops away (Dual XEON) that got over loaded the most. Our routers have been tested to pass over 2 gbps of throughput easilly. And have been load tested to survive very small packets and high PPS adequately. The infected sub was bandwidth managed with HTB to 256k cir, 1 mbps mir, but not anything for PPS. So I'm looking for reasons that the CPU got overloaded. My theory is that the DOS attack resulted in a large number of disk writes, ( maybe logging?) causing the CPU saturation. I've had a hard time locating the cause. And have not discovered which virus yet, although I should have more info soon from my clients. So my question What needs to be done on a Linux machine to harden it, to protect against CPU oversaturation, during DOS attacks? What should and shouldn't be logged? Connection Tracking? Firewall logging? Traffic stats? Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] SSH DOS Killing Linux
- Original Message - From: Tom DeReggi [mailto:[EMAIL PROTECTED] To: WISPA General List [mailto:[EMAIL PROTECTED] Sent: Sat, 06 Jan 2007 17:26:39 -0900 Subject: [WISPA] SSH DOS Killing Linux We recently had a really nasty DOS attack that took down a large part of our network across several cell sites, from the infected client all the way to the Internet transit. Take note that we identified the problem quickly and cured it quickly. But This is the first time that this has occured in 5 years, as we have a good number of smart design characteristics that have limited the effects of most viruses on our network. We stopped the attack, by blocking SSH to the infected sub. The average amount of traffic crossing the entire network path from the client to the Internet was about 500 kbps on average. (This was a 20 mbps wireless link, and a 100mbps fiber trnasport link to the transit.). The two routers were a P4 2Ghz, and a Dual XEON 2.2Ghz w/ 10,000rpm SCSI3. The damage was that the CPU was nailed on both routers to about 99.9% using TOP to monitor stats. We varified that successful SSH sessions were not made directly to the protected routers themselves. Take note that the wireless links were barely effected, it was the router 2 hops away (Dual XEON) that got over loaded the most. Our routers have been tested to pass over 2 gbps of throughput easilly. And have been load tested to survive very small packets and high PPS adequately. The infected sub was bandwidth managed with HTB to 256k cir, 1 mbps mir, but not anything for PPS. So I'm looking for reasons that the CPU got overloaded. My theory is that the DOS attack resulted in a large number of disk writes, ( maybe logging?) causing the CPU saturation. I've had a hard time locating the cause. And have not discovered which virus yet, although I should have more info soon from my clients. So my question What needs to be done on a Linux machine to harden it, to protect against CPU oversaturation, during DOS attacks? What should and shouldn't be logged? Connection Tracking? Firewall logging? Traffic stats? Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband Hi Tom, What OS/application was running on these boxes? -Dee Alaska Wireless Systems 1(907)240-2183 Cell 1(907)349-2226 Fax 1(907)349-4308 Office www.akwireless.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
