Re: Defending against the "Apache killer"
by the way, you can find some HAProxy configuration to fight against this attack on our blog: http://blog.exceliance.fr/2011/08/25/protect-apache-against-apache-killer-script/ cheers
Re: Defending against the "Apache killer"
On Wed, Aug 24, 2011 at 1:44 PM, Cyril Bonté wrote:
> Hi all,
>
> On Wednesday 24 August 2011 13:02:18 Baptiste wrote:
> (...)
>> > Since there is no hdr_size ACLs for now, the only way is to use a
>> > hdr_reg to do this:
>> > reqidel ^Range if { hdr_reg(Range) ([0-9]+-[0-9]+,){10,} }
>> >
>> > But the regexp above does not work (haproxy 1.5-dev6), the comma is
>> > not matched
>> > don't know yet if it's an haproxy bug or not, I'll tell you once I
>> > have finished investigating.
>> >
>> > cheers
>>
>> I confirm, this looks like a bug in HAProxy, maybe in the way HAProxy
>> loads the regexp from the configuration file:
>
> This is not how HAProxy loads the regex but how it applies them to the
> headers.
> The comma character (,) is considered as a value separator. HAProxy will then
> try to apply the regex to each value found in the Range header.
> For this header :
> Range:
> bytes=0-,5-0,5-1,5-2,5-3,5-4,5-5,5-6,5-7,5-8,5-9,5-10,5-11,5-12,5-13,5-14,5-15,5-16,5-17,5-18,5-19,5-20,5-21,5-22,5-23,5-24,5-25,5-26,5-27,5-28,5-29,5-30,5-31,5-32,5-33,5-34,5-35,5-36,5-37,5-38,5-39,5-40,5-41,5-42,5-43,5-44,5-45,5-46,5-47,5-48,5-49,5-50,5-51,5-52,5-53,5-54,5-55,5-56,5-57,5-58,5-59,5-60,5-61,5-62,5-63,5-64,5-65,5-66,5-67,5-68,5-69,5-70,5-71,5-72,5-73,5-74,5-75,5-76,5-77,5-78,5-79,5-80,5-81,5-82,5-83,5-84,5-85,5-86,5-87,5-88,5-89,5-90,5-91,5-92,5-93,5-94,5-95,5-96,5-97
>
> It will check byte=0-
> then 5-0
> then 5-1
> then ...
>
>
> --
> Cyril Bonté
>
ahah :)
You're both all right.
Sorry, I totaly forgot this part of the RFC:
"Multiple message-header fields with the same field-name MAY be
present in a message if and only if the entire field-value for that
header field is defined as a comma-separated list [i.e., #(values)].
It MUST be possible to combine the multiple header fields into one
"field-name: field-value" pair, without changing the semantics of the
message, by appending each subsequent field-value to the first, each
separated by a comma. The order in which header fields with the same
field-name are received is therefore significant to the interpretation
of the combined field value, and thus a proxy MUST NOT change the
order of these field values when a message is forwarded.
"
So the hdr_cnt from willy works.
I did not try this option since this is not how the Perl script of the
first mail build the attack.
sorry for the noise and glad to see tha HAProxy works well :)
Re: Defending against the "Apache killer"
Hi all,
On Wednesday 24 August 2011 13:02:18 Baptiste wrote:
(...)
> > Since there is no hdr_size ACLs for now, the only way is to use a
> > hdr_reg to do this:
> > reqidel ^Range if { hdr_reg(Range) ([0-9]+-[0-9]+,){10,} }
> >
> > But the regexp above does not work (haproxy 1.5-dev6), the comma is
> > not matched
> > don't know yet if it's an haproxy bug or not, I'll tell you once I
> > have finished investigating.
> >
> > cheers
>
> I confirm, this looks like a bug in HAProxy, maybe in the way HAProxy
> loads the regexp from the configuration file:
This is not how HAProxy loads the regex but how it applies them to the
headers.
The comma character (,) is considered as a value separator. HAProxy will then
try to apply the regex to each value found in the Range header.
For this header :
Range:
bytes=0-,5-0,5-1,5-2,5-3,5-4,5-5,5-6,5-7,5-8,5-9,5-10,5-11,5-12,5-13,5-14,5-15,5-16,5-17,5-18,5-19,5-20,5-21,5-22,5-23,5-24,5-25,5-26,5-27,5-28,5-29,5-30,5-31,5-32,5-33,5-34,5-35,5-36,5-37,5-38,5-39,5-40,5-41,5-42,5-43,5-44,5-45,5-46,5-47,5-48,5-49,5-50,5-51,5-52,5-53,5-54,5-55,5-56,5-57,5-58,5-59,5-60,5-61,5-62,5-63,5-64,5-65,5-66,5-67,5-68,5-69,5-70,5-71,5-72,5-73,5-74,5-75,5-76,5-77,5-78,5-79,5-80,5-81,5-82,5-83,5-84,5-85,5-86,5-87,5-88,5-89,5-90,5-91,5-92,5-93,5-94,5-95,5-96,5-97
It will check byte=0-
then 5-0
then 5-1
then ...
--
Cyril Bonté
Re: Defending against the "Apache killer"
On Wed, Aug 24, 2011 at 12:44:45PM +0200, Baptiste wrote:
>
> [...]
> Actually, this is slightly different.
> According to the Perl script, a single Range header is sent, but it is
> forge with a lot of range value.
> IE: "Range: 0-,5-1,5-2,5-3,[...]"
>
> Since there is no hdr_size ACLs for now, the only way is to use a
> hdr_reg to do this:
> reqidel ^Range if { hdr_reg(Range) ([0-9]+-[0-9]+,){10,} }
HTTP headers don't work like that. Range: 0-,5-,... is the same as
Range: 0-
Range: 5-
Range: ...
So hdr_cnt() should pick it up just fine, and the regex should not.
- Finn Arne
Re: Defending against the "Apache killer"
On Wed, Aug 24, 2011 at 12:44 PM, Baptiste wrote:
> On Tue, Aug 23, 2011 at 8:09 AM, Willy Tarreau wrote:
>> On Mon, Aug 22, 2011 at 07:57:10PM +0200, Baptiste wrote:
>>> Hi,
>>>
>>> Why not only dropping this "Range:bytes=0-" header?
>>
>> Agreed. Protecting against this vulnerability is not a matter of limiting
>> connections or whatever. The attack makes mod_deflate exhaust the process'
>> memory. What is needed is to remove the Range header when there are too
>> many occurrences of it.
>>
>> Their attack puts up to 1300 Range values. Let's remove the header if
>> there are more than 2 :
>>
>> reqidel ^Range if { hdr_cnt(Range) gt 2 }
>>
>> That should reliably defeat the attack.
>>
>> Regards,
>> Willy
>>
>>
>
>
> Actually, this is slightly different.
> According to the Perl script, a single Range header is sent, but it is
> forge with a lot of range value.
> IE: "Range: 0-,5-1,5-2,5-3,[...]"
>
> Since there is no hdr_size ACLs for now, the only way is to use a
> hdr_reg to do this:
> reqidel ^Range if { hdr_reg(Range) ([0-9]+-[0-9]+,){10,} }
>
> But the regexp above does not work (haproxy 1.5-dev6), the comma is
> not matched
> don't know yet if it's an haproxy bug or not, I'll tell you once I
> have finished investigating.
>
> cheers
>
I confirm, this looks like a bug in HAProxy, maybe in the way HAProxy
loads the regexp from the configuration file:
Here is a req.txt file simulating the attack:
HEAD / HTTP/1.1
Host: 10.0.3.20
Range:
bytes=0-,5-0,5-1,5-2,5-3,5-4,5-5,5-6,5-7,5-8,5-9,5-10,5-11,5-12,5-13,5-14,5-15,5-16,5-17,5-18,5-19,5-20,5-21,5-22,5-23,5-24,5-25,5-26,5-27,5-28,5-29,5-30,5-31,5-32,5-33,5-34,5-35,5-36,5-37,5-38,5-39,5-40,5-41,5-42,5-43,5-44,5-45,5-46,5-47,5-48,5-49,5-50,5-51,5-52,5-53,5-54,5-55,5-56,5-57,5-58,5-59,5-60,5-61,5-62,5-63,5-64,5-65,5-66,5-67,5-68,5-69,5-70,5-71,5-72,5-73,5-74,5-75,5-76,5-77,5-78,5-79,5-80,5-81,5-82,5-83,5-84,5-85,5-86,5-87,5-88,5-89,5-90,5-91,5-92,5-93,5-94,5-95,5-96,5-97
Accept-Encoding: gzip
Connection: close
And a working regexp tested with egrep:
egrep -v "([0-9]+-[0-9]+,){10,}" req.txt
HEAD / HTTP/1.1
Host: 10.0.3.20
Accept-Encoding: gzip
Connection: close
The following regexp works in HAProxy: ([0-9]+-[0-9]+)
The same with the coma does not work: ([0-9]+-[0-9]+,)
This one works: ([0-9]+-[0-9]+?)
And this one does not: ([0-9]+-[0-9]+?)\{10,\}
Maybe I'm doing something wrong.
If your need more details, please let me know.
cheers
Re: Defending against the "Apache killer"
On Tue, Aug 23, 2011 at 8:09 AM, Willy Tarreau wrote:
> On Mon, Aug 22, 2011 at 07:57:10PM +0200, Baptiste wrote:
>> Hi,
>>
>> Why not only dropping this "Range:bytes=0-" header?
>
> Agreed. Protecting against this vulnerability is not a matter of limiting
> connections or whatever. The attack makes mod_deflate exhaust the process'
> memory. What is needed is to remove the Range header when there are too
> many occurrences of it.
>
> Their attack puts up to 1300 Range values. Let's remove the header if
> there are more than 2 :
>
> reqidel ^Range if { hdr_cnt(Range) gt 2 }
>
> That should reliably defeat the attack.
>
> Regards,
> Willy
>
>
Actually, this is slightly different.
According to the Perl script, a single Range header is sent, but it is
forge with a lot of range value.
IE: "Range: 0-,5-1,5-2,5-3,[...]"
Since there is no hdr_size ACLs for now, the only way is to use a
hdr_reg to do this:
reqidel ^Range if { hdr_reg(Range) ([0-9]+-[0-9]+,){10,} }
But the regexp above does not work (haproxy 1.5-dev6), the comma is
not matched
don't know yet if it's an haproxy bug or not, I'll tell you once I
have finished investigating.
cheers
Re: Defending against the "Apache killer"
On Mon, Aug 22, 2011 at 07:57:10PM +0200, Baptiste wrote:
> Hi,
>
> Why not only dropping this "Range:bytes=0-" header?
Agreed. Protecting against this vulnerability is not a matter of limiting
connections or whatever. The attack makes mod_deflate exhaust the process'
memory. What is needed is to remove the Range header when there are too
many occurrences of it.
Their attack puts up to 1300 Range values. Let's remove the header if
there are more than 2 :
reqidel ^Range if { hdr_cnt(Range) gt 2 }
That should reliably defeat the attack.
Regards,
Willy
Re: Defending against the "Apache killer"
On Mon, Aug 22, 2011 at 06:26:01PM +, Svancara, Randall wrote: > This is nothing new as brute force DOS attacks have been around for a while. > I am not sure this is an HA-Proxy feature or more of a MOD_SECURITY/iptables > feature. Simple iptables rate limiting would be sufficient in thwarting this > attack. For example, > > I am using this for SSH now, but very applicable to a web server, change the > ports and hitcount to a number that is more appropriate for a webserver, like > 40 in 10 seconds. > > # Drop those nasty brute force SSH attempts and log them > $IPTABLES -A INPUT -p tcp --dport 22 -i $EXTIF -m state --state NEW -m recent > --set > $IPTABLES -A INPUT -p tcp --dport 22 -i $EXTIF -m state --state NEW -m > recent --update --seconds 60 --hitcount 4 -j SSHBRUTEDROP > > I am using the above code to block ssh brute force attempts. Doing so is already possible with haproxy but this has nothing to do with the attack, as it's not a matter of request rate but memory exhaustion on the server due to a vulnerability. Regards, Willy
RE: Defending against the "Apache killer"
This is nothing new as brute force DOS attacks have been around for a while. I am not sure this is an HA-Proxy feature or more of a MOD_SECURITY/iptables feature. Simple iptables rate limiting would be sufficient in thwarting this attack. For example, I am using this for SSH now, but very applicable to a web server, change the ports and hitcount to a number that is more appropriate for a webserver, like 40 in 10 seconds. # Drop those nasty brute force SSH attempts and log them $IPTABLES -A INPUT -p tcp --dport 22 -i $EXTIF -m state --state NEW -m recent --set $IPTABLES -A INPUT -p tcp --dport 22 -i $EXTIF -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j SSHBRUTEDROP I am using the above code to block ssh brute force attempts. --Randall -Original Message- From: Levente Peres [mailto:sheri...@eurosystems.hu] Sent: Monday, August 22, 2011 9:54 AM To: haproxy@formilux.org Subject: Defending against the "Apache killer" Hello, There're a number of webserver-mace apps on the net, the newest that I heard of being the so called "Apache killer" script I saw a few days agon on Full disclosure... Here you can see a demonstration of what it does. Also, I've attached the script itself. http://www.youtube.com/watch?v=fkCQZaVjBhA I believe we should discuss some possibilities about how to configure HAProxy to protect Apache backends as much as possible, or at least mitigate such attacks? Any ideas? Cheers, Levente
Re: Defending against the "Apache killer"
Hi, 1. install nginx as frontend 2. install latest version of Apache as backend (afair 2.2.18 was not vulnerable to such DoS already, and 2.2.19 should be ok too) 3. remove apache's mod_deflate 4. done -- Cheers, Kai
Re: Defending against the "Apache killer"
Hi, Why not only dropping this "Range:bytes=0-" header? cheers 2011/8/22 Levente Peres : > Hello, > > There're a number of webserver-mace apps on the net, the newest that I heard > of being the so called "Apache killer" script I saw a few days agon on Full > disclosure... Here you can see a demonstration of what it does. Also, I've > attached the script itself. > > http://www.youtube.com/watch?v=fkCQZaVjBhA > > I believe we should discuss some possibilities about how to configure > HAProxy to protect Apache backends as much as possible, or at least mitigate > such attacks? Any ideas? > > Cheers, > > Levente >
