Re: [Owasp-modsecurity-core-rule-set] Substantial and unacceptable latency impact using mod_security and core rule set

[Christian Folini]

Hello Mark,

On Thu, Feb 01, 2018 at 09:27:13PM +, Mark Blackman wrote:
> Thanks, as an update, a second round of testing where logging was reduced
> and where we used a more proven httpd configuration resulted in more
> sensible results, typically 2 ms for a request without scanning and 4 ms for
> a request with scanning.

That is good news and I think also realistic. You can bring it further down
with tuning. But it takes some effort.

> We’re using version 2.9, but I wonder how much improvement you might expect
> for version 3.0 and is version 3.0 considered production ready yet?

ModSec 3.0 has been released as production ready. CRS3 works and namely
the NGINX Plus offering has been using it for quite some time.

The developers claim a seed gain over ModSec 2.9.x yet there are also people
talking of performance issues. Personally, I do not have trustworthy data,
so I do not want to add to any rumours one way or the other. I think it is
best to test for yourself for the time being. Also I expect further
optimizations with the 3.0.1 release.

> Finally, does mod_security use any kind of hashing to cache results?

No, it does not. At least 2.9 does not and I do not think I have read of
3.0 bringing this. It's an interesting feature, but also one that is very
hard to get right, namely as this is a security product and hashing/caching
often get into a conflict with one another.

> On the
> assumption that computing a digest of request is faster than running a large
> series of regexes over it. In other words, there’s no point repeatedly
> running the same rules over the same request line when it comes in each
> time. Alternately, can mod_security exploit Apache server-side caching to
> cache results?

Again, no.

However, if you really want to speed things up and you have a good
understanding of your application, authentication is in place and you really
know whom you are talking to, then you could think of bypassing certain
rules under certain conditions. That is a trade-off between security and
speed, but given the complete rule language is at your disposal interesting
constructs can be created.

Cheers,

Christian


-- 
Human beings, who are almost unique in having the ability to learn
from the experience of others, are also remarkable for their apparent
disinclination to do so.
--- Douglas Adams
___
Owasp-modsecurity-core-rule-set mailing list
Owasp-modsecurity-core-rule-set@lists.owasp.org
https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set

Re: [Owasp-modsecurity-core-rule-set] Substantial and unacceptable latency impact using mod_security and core rule set

[Mark Blackman]

> On 18 Dec 2017, at 13:41, Christian Folini  
> wrote:
> 
> Mark,
> 
> Latency is an issue and the amount depends on the server. Factor 5 is a bit
> steep, but still possible.
> 
> My mileage is usually a 5-10% hit on the throughput of a reverse proxy. If
> your server serves only static files and no backend connection, then your
> numbers could be real.
> 
> I would want to look at the individual requests, though.
> Is 40 ms the mean? Is it some requests or all of them? If some, which ones?
> 
> The tutorials at https://www.netnea.com come with an extended Apache Access
> Log format that lets you gauge the performance impact a bit better. Once you
> have the data, you can dig down and identify the individual requests / rules
> that cause the delay. Raising the debug-log-level on individual requests let
> you identify the individual rule of an individual request with a high
> performance impact quite easily. And then tune them away.
> 
> Also, CRS3 comes with a feature called Sampling Mode that allows you to run
> the rules only on a percentage of the requests. This allows you to test / tune
> with real world data without bringing the whole service down. This is
> specifically aimed at services where the performance impact is unclear
> and a potential risk.
> 
> Performance issues are generally solvable with a compromise between
> performance and security. And ModSec gives you a ton of performance
> information so it is generally possible to nail down the problem.
> 
> Hope this helps for a start. If you need more infos, then just ask.

Thanks, as an update, a second round of testing where logging was reduced and 
where we used a more proven httpd configuration resulted in more sensible 
results, typically 2 ms for a request without scanning and 4 ms for a request 
with scanning.

We’re using version 2.9, but I wonder how much improvement you might expect for 
version 3.0 and is version 3.0 considered production ready yet?

Finally, does mod_security use any kind of hashing to cache results? On the 
assumption that computing a digest of request is faster than running a large 
series of regexes over it. In other words, there’s no point repeatedly running 
the same rules over the same request line when it comes in each time. 
Alternately, can mod_security exploit Apache server-side caching to cache 
results?

Regards,
Mark


___
Owasp-modsecurity-core-rule-set mailing list
Owasp-modsecurity-core-rule-set@lists.owasp.org
https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set

Re: [Owasp-modsecurity-core-rule-set] Substantial and unacceptable latency impact using mod_security and core rule set

[Christian Folini]

Mark,

Latency is an issue and the amount depends on the server. Factor 5 is a bit
steep, but still possible.

My mileage is usually a 5-10% hit on the throughput of a reverse proxy. If
your server serves only static files and no backend connection, then your
numbers could be real.

I would want to look at the individual requests, though.
Is 40 ms the mean? Is it some requests or all of them? If some, which ones?

The tutorials at https://www.netnea.com come with an extended Apache Access
Log format that lets you gauge the performance impact a bit better. Once you
have the data, you can dig down and identify the individual requests / rules
that cause the delay. Raising the debug-log-level on individual requests let
you identify the individual rule of an individual request with a high
performance impact quite easily. And then tune them away.

Also, CRS3 comes with a feature called Sampling Mode that allows you to run
the rules only on a percentage of the requests. This allows you to test / tune
with real world data without bringing the whole service down. This is
specifically aimed at services where the performance impact is unclear
and a potential risk.

Performance issues are generally solvable with a compromise between
performance and security. And ModSec gives you a ton of performance
information so it is generally possible to nail down the problem.

Hope this helps for a start. If you need more infos, then just ask.

Good luck!

Christian

On Mon, Dec 18, 2017 at 01:24:48PM +, Mark Blackman wrote:
> Hi,
> 
> In an initial set of performance tests with mod_security 2.9.2 and the core
> rule set under Apache 2.2.34, we are seeing the latency for individual
> requests rise from 8 milliseconds to 40 milliseconds which seems like too
> much.   Is this kind of latency impact to be expected? What options do we
> have for reducing it without throwing away mod_security?
> 
> - Mark ___
> Owasp-modsecurity-core-rule-set mailing list
> Owasp-modsecurity-core-rule-set@lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set

-- 
https://www.feistyduck.com/training/modsecurity-training-course
https://www.feistyduck.com/books/modsecurity-handbook/
mailto:christian.fol...@netnea.com
twitter: @ChrFolini
___
Owasp-modsecurity-core-rule-set mailing list
Owasp-modsecurity-core-rule-set@lists.owasp.org
https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set

[Owasp-modsecurity-core-rule-set] Substantial and unacceptable latency impact using mod_security and core rule set

[Mark Blackman]

Hi,

In an initial set of performance tests with mod_security 2.9.2 and the core 
rule set under Apache 2.2.34, we are seeing the latency for individual requests 
rise from 8 milliseconds to 40 milliseconds which seems like too much.   Is 
this kind of latency impact to be expected? What options do we have for 
reducing it without throwing away mod_security?

- Mark
___
Owasp-modsecurity-core-rule-set mailing list
Owasp-modsecurity-core-rule-set@lists.owasp.org
https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set