Re: Defending against the Apache killer
On Tue, Aug 23, 2011 at 8:09 AM, Willy Tarreau w...@1wt.eu wrote:
On Mon, Aug 22, 2011 at 07:57:10PM +0200, Baptiste wrote:
Hi,
Why not only dropping this Range:bytes=0- header?
Agreed. Protecting against this vulnerability is not a matter of limiting
connections or whatever. The attack makes mod_deflate exhaust the process'
memory. What is needed is to remove the Range header when there are too
many occurrences of it.
Their attack puts up to 1300 Range values. Let's remove the header if
there are more than 2 :
reqidel ^Range if { hdr_cnt(Range) gt 2 }
That should reliably defeat the attack.
Regards,
Willy
Actually, this is slightly different.
According to the Perl script, a single Range header is sent, but it is
forge with a lot of range value.
IE: Range: 0-,5-1,5-2,5-3,[...]
Since there is no hdr_size ACLs for now, the only way is to use a
hdr_reg to do this:
reqidel ^Range if { hdr_reg(Range) ([0-9]+-[0-9]+,){10,} }
But the regexp above does not work (haproxy 1.5-dev6), the comma is
not matched
don't know yet if it's an haproxy bug or not, I'll tell you once I
have finished investigating.
cheers
Re: Defending against the Apache killer
On Wed, Aug 24, 2011 at 12:44 PM, Baptiste bed...@gmail.com wrote:
On Tue, Aug 23, 2011 at 8:09 AM, Willy Tarreau w...@1wt.eu wrote:
On Mon, Aug 22, 2011 at 07:57:10PM +0200, Baptiste wrote:
Hi,
Why not only dropping this Range:bytes=0- header?
Agreed. Protecting against this vulnerability is not a matter of limiting
connections or whatever. The attack makes mod_deflate exhaust the process'
memory. What is needed is to remove the Range header when there are too
many occurrences of it.
Their attack puts up to 1300 Range values. Let's remove the header if
there are more than 2 :
reqidel ^Range if { hdr_cnt(Range) gt 2 }
That should reliably defeat the attack.
Regards,
Willy
Actually, this is slightly different.
According to the Perl script, a single Range header is sent, but it is
forge with a lot of range value.
IE: Range: 0-,5-1,5-2,5-3,[...]
Since there is no hdr_size ACLs for now, the only way is to use a
hdr_reg to do this:
reqidel ^Range if { hdr_reg(Range) ([0-9]+-[0-9]+,){10,} }
But the regexp above does not work (haproxy 1.5-dev6), the comma is
not matched
don't know yet if it's an haproxy bug or not, I'll tell you once I
have finished investigating.
cheers
I confirm, this looks like a bug in HAProxy, maybe in the way HAProxy
loads the regexp from the configuration file:
Here is a req.txt file simulating the attack:
HEAD / HTTP/1.1
Host: 10.0.3.20
Range:
bytes=0-,5-0,5-1,5-2,5-3,5-4,5-5,5-6,5-7,5-8,5-9,5-10,5-11,5-12,5-13,5-14,5-15,5-16,5-17,5-18,5-19,5-20,5-21,5-22,5-23,5-24,5-25,5-26,5-27,5-28,5-29,5-30,5-31,5-32,5-33,5-34,5-35,5-36,5-37,5-38,5-39,5-40,5-41,5-42,5-43,5-44,5-45,5-46,5-47,5-48,5-49,5-50,5-51,5-52,5-53,5-54,5-55,5-56,5-57,5-58,5-59,5-60,5-61,5-62,5-63,5-64,5-65,5-66,5-67,5-68,5-69,5-70,5-71,5-72,5-73,5-74,5-75,5-76,5-77,5-78,5-79,5-80,5-81,5-82,5-83,5-84,5-85,5-86,5-87,5-88,5-89,5-90,5-91,5-92,5-93,5-94,5-95,5-96,5-97
Accept-Encoding: gzip
Connection: close
And a working regexp tested with egrep:
egrep -v ([0-9]+-[0-9]+,){10,} req.txt
HEAD / HTTP/1.1
Host: 10.0.3.20
Accept-Encoding: gzip
Connection: close
The following regexp works in HAProxy: ([0-9]+-[0-9]+)
The same with the coma does not work: ([0-9]+-[0-9]+,)
This one works: ([0-9]+-[0-9]+?)
And this one does not: ([0-9]+-[0-9]+?)\{10,\}
Maybe I'm doing something wrong.
If your need more details, please let me know.
cheers
Re: Defending against the Apache killer
Hi all,
On Wednesday 24 August 2011 13:02:18 Baptiste wrote:
(...)
Since there is no hdr_size ACLs for now, the only way is to use a
hdr_reg to do this:
reqidel ^Range if { hdr_reg(Range) ([0-9]+-[0-9]+,){10,} }
But the regexp above does not work (haproxy 1.5-dev6), the comma is
not matched
don't know yet if it's an haproxy bug or not, I'll tell you once I
have finished investigating.
cheers
I confirm, this looks like a bug in HAProxy, maybe in the way HAProxy
loads the regexp from the configuration file:
This is not how HAProxy loads the regex but how it applies them to the
headers.
The comma character (,) is considered as a value separator. HAProxy will then
try to apply the regex to each value found in the Range header.
For this header :
Range:
bytes=0-,5-0,5-1,5-2,5-3,5-4,5-5,5-6,5-7,5-8,5-9,5-10,5-11,5-12,5-13,5-14,5-15,5-16,5-17,5-18,5-19,5-20,5-21,5-22,5-23,5-24,5-25,5-26,5-27,5-28,5-29,5-30,5-31,5-32,5-33,5-34,5-35,5-36,5-37,5-38,5-39,5-40,5-41,5-42,5-43,5-44,5-45,5-46,5-47,5-48,5-49,5-50,5-51,5-52,5-53,5-54,5-55,5-56,5-57,5-58,5-59,5-60,5-61,5-62,5-63,5-64,5-65,5-66,5-67,5-68,5-69,5-70,5-71,5-72,5-73,5-74,5-75,5-76,5-77,5-78,5-79,5-80,5-81,5-82,5-83,5-84,5-85,5-86,5-87,5-88,5-89,5-90,5-91,5-92,5-93,5-94,5-95,5-96,5-97
It will check byte=0-
then 5-0
then 5-1
then ...
--
Cyril Bonté
Re: Defending against the Apache killer
On Wed, Aug 24, 2011 at 1:44 PM, Cyril Bonté cyril.bo...@free.fr wrote:
Hi all,
On Wednesday 24 August 2011 13:02:18 Baptiste wrote:
(...)
Since there is no hdr_size ACLs for now, the only way is to use a
hdr_reg to do this:
reqidel ^Range if { hdr_reg(Range) ([0-9]+-[0-9]+,){10,} }
But the regexp above does not work (haproxy 1.5-dev6), the comma is
not matched
don't know yet if it's an haproxy bug or not, I'll tell you once I
have finished investigating.
cheers
I confirm, this looks like a bug in HAProxy, maybe in the way HAProxy
loads the regexp from the configuration file:
This is not how HAProxy loads the regex but how it applies them to the
headers.
The comma character (,) is considered as a value separator. HAProxy will then
try to apply the regex to each value found in the Range header.
For this header :
Range:
bytes=0-,5-0,5-1,5-2,5-3,5-4,5-5,5-6,5-7,5-8,5-9,5-10,5-11,5-12,5-13,5-14,5-15,5-16,5-17,5-18,5-19,5-20,5-21,5-22,5-23,5-24,5-25,5-26,5-27,5-28,5-29,5-30,5-31,5-32,5-33,5-34,5-35,5-36,5-37,5-38,5-39,5-40,5-41,5-42,5-43,5-44,5-45,5-46,5-47,5-48,5-49,5-50,5-51,5-52,5-53,5-54,5-55,5-56,5-57,5-58,5-59,5-60,5-61,5-62,5-63,5-64,5-65,5-66,5-67,5-68,5-69,5-70,5-71,5-72,5-73,5-74,5-75,5-76,5-77,5-78,5-79,5-80,5-81,5-82,5-83,5-84,5-85,5-86,5-87,5-88,5-89,5-90,5-91,5-92,5-93,5-94,5-95,5-96,5-97
It will check byte=0-
then 5-0
then 5-1
then ...
--
Cyril Bonté
ahah :)
You're both all right.
Sorry, I totaly forgot this part of the RFC:
Multiple message-header fields with the same field-name MAY be
present in a message if and only if the entire field-value for that
header field is defined as a comma-separated list [i.e., #(values)].
It MUST be possible to combine the multiple header fields into one
field-name: field-value pair, without changing the semantics of the
message, by appending each subsequent field-value to the first, each
separated by a comma. The order in which header fields with the same
field-name are received is therefore significant to the interpretation
of the combined field value, and thus a proxy MUST NOT change the
order of these field values when a message is forwarded.
So the hdr_cnt from willy works.
I did not try this option since this is not how the Perl script of the
first mail build the attack.
sorry for the noise and glad to see tha HAProxy works well :)
Re: X-Forwarded-For contortions
I think this will address my issue, thanks a lot! Sorry for not getting back to your questions sooner; I just returned from an end-of-summer trip. I'll check it out soon. It's in 1.4 trunk but not in any release yet, correct? -Bryan On Fri, Aug 19, 2011 at 2:04 PM, Willy Tarreau w...@1wt.eu wrote: Hi Bryan, On Tue, Aug 16, 2011 at 07:51:07AM +0200, Willy Tarreau wrote: Hi Bryan, On Mon, Aug 15, 2011 at 11:13:18AM -0700, Bryan Talbot wrote: That would work but there are several CIDR networks that contain a trusted proxy. The CDN is global and has proxies on most continents and many of them are on different networks. Since the option forwardfor can only handle a single network in the if|unless argument the only real use case it can support is where all the upstream proxies come from the same network -- like for an upstream SSL terminator. The logic I need is: if traffic is from a trusted upstream proxy of multiple networks leave existing XFF header alone else strip any XFF header from request and add our own Well, I think we could see it slightly differently : if traffic is from a trusted upstream proxy of multiple networks leave existing XFF header alone else strip any XFF header from request if (no XFF is present) and add our own This way we could use the ACLs to delete any existing occurrence and have a simple flag to indicate the addition is conditionnal. For instance we could have a dont-replace flag on the fwdfor config line to indicate that it should only be set and not replaced. However we have to keep in mind that XFF is performed both in the frontend and the backend, so the conditional logic must be compatible between both. This means that if either of the front or back has the option without the dont-replace flag, it will still be replaced. I did it, though I changed the argument name to if-none to make it less ambiguous especially in the case where both the frontend and the backend have the option. It is even supported in the defaults section (but this should be avoided since it has security implications). Thus, for your case above, you'd simply remove the header if the IP comes from an untrusted host, then conditionally add it : reqidel ^x-forwarded-for if !from_proxies option forwardfor if-none I'm attaching the patch, but it's already pushed upstream. Regards, Willy
Send frontend response before getting response from remote web server
Hi, We have some HTTP calls from our app that we'd like to fire and forget, but we need the Haproxy frontend send a response to the requester immediately after it passes the request to the backend queue. The intention is to replace our Gearman setup and thus save us some time in maintenance and management as we can then just deal in HTTP calls, it also saves us having to write PHP scripts for gearman processing. What we need is to just be able to execute a curl call and send a response back as as soon as possible so the curl call can complete and the script can finish processing. The chain we want is: 1. Local webserver --- request --- Haproxy frontend 2. Local webserver --- local OK response --- Haproxy frontend --- request (queue) --- Haproxy backend 3. Haproxy backend --- request --- Remote webserver 4. Haproxy backend --- remote response --- Remote webserver 5. dumped --- remote response --- Haproxy backend We did some research and it seems like both of the following are possible: 1. Send an automatic HTTP response code from the frontend to the requesting web server. 2. Put the request in the backend queue to be sent to the remote server However, with the above it doesn't seem like we can send the response from the frontend *immediately* after we put the request in the backend queue. I hope that makes sense. Can anyone provide any feedback on whether something like this is possible? Thanks, Guy -- Guy Knights Systems Administrator Eastside Games www.eastsidegamestudio.com g...@eastsidegamestudio.com
