On Wed, 2018-12-12 at 16:22:35 +0100, Aksel Cevdet Devrimci wrote:
Turned off TLS and it dropped to 10% ish so looks like gnutls might be the
problem. AES-NI is active on the VM(tested with openssl), but not sure how to
test with gnutls.
Tried to set the env variable
GNUTLS_CPUID_OVERRIDE(https://www.gnutls.org/manual/gnutls.html#gnutls_005fglobal_005finit)
to both off and on but there where no difference. Same goes when we disable
AES-NI support on the VM. Still stuck at 5,8K while using relp with gnutls.
Ah, can you check how the package you're using is built? There appears
to be options to turn off hardware acceleration
(--disable-hardware-acceleration and --disable-padlock) and supposedly
to turn it on --enable-cryptodev.
Tried with some different options with the tls.prioritystring but had
no effect.
As you can see with `gnutls-cli --benchmark-tls-kx` the default RSA is
pretty slow (in my case 4 times slower) and as you said, disabling or
enabling crypto acc makes no difference for you but there should be a
difference between:
`env GNUTLS_CPUID_OVERRIDE=0x1 gnutls-cli --benchmark-tls-ciphers`
and
`gnutls-cli --benchmark-tls-ciphers`
Please check with `gnutls-cli --priority 'PRIORITY:STRING' -l` that you
actually got what you expected.
BR,
- Simon
________________________________
Fra: Simon Lundström <[email protected]>
Sendt: onsdag 12. desember 2018 13.59.00
Til: Aksel Cevdet Devrimci
Kopi: [email protected]
Emne: Re: [rsyslog] CPU at 100% while using imrelp.
On Wed, 2018-12-12 at 12:30:05 +0100, Aksel Cevdet Devrimci wrote:
As you can see it's the imrelp thread that is always at a 100%
And you said so in your first mail, I missed that sorry.
Can you push more than 5.8k EPS through rsyslog even though it's using
100% CPU?
Upgrading librelp and rsyslog to the latest version did not have any effect.
Is it possible to have multiple imrelp threads?
Doesn't look like it, someone from Adiscon will have to answer that
though.
Can you, as a test, turn off TLS to see if it's the decryption which
uses CPU? If not this might be a bug in RELP?
If it's the decryption which uses a lot of CPU:
* Is it a VM? Does the VM have access to the HW crypto accelerator? Is
GnuTLS using the crypto acc? (I don't know how to check this).
* Are the clients doing a lot of short sessions? Can you make sure they
are using very long connection sessions? RSA, which I'm guessing is the
default (from
<https://github.com/rsyslog/librelp/blob/bd6c0a7fc7dd2dbedf772c1de8a500ad602a8c97/src/tcp.c#L1144>
and <https://gnutls.org/manual/html_node/Priority-Strings.html>), is CPU
intensive so long living sessions are prefered.
* ECDSA should be faster than RSA, you can set it via
<https://www.rsyslog.com/doc/v8-stable/configuration/modules/imrelp.html#tls-prioritystring>
and <https://gnutls.org/manual/html_node/Priority-Strings.html>
Basically everything that applies to making HTTPS fast applies here as
well.
Good luck and please report back of your, hopefully, great success = )
BR,
- Simon
________________________________
Fra: Simon Lundström <[email protected]>
Sendt: onsdag 12. desember 2018 10.53.59
Til: rsyslog-users
Kopi: Aksel Cevdet Devrimci
Emne: Re: [rsyslog] CPU at 100% while using imrelp.
Be sure to check what is using CPU too. If you're using Linux and GNU
top you can use `top -H -p $(pgrep rsyslog)` to see the threads (or
press H after you've started top IIRC).
Each module has one or multiple threaded workers so you can easily see
what is using CPU.
BR,
- Simon
On Wed, 2018-12-12 at 09:36:18 +0100, Aksel Cevdet Devrimci via rsyslog wrote:
No my bad, is ofc 8.39 :)
Now that rsyslog 8.40 and librelp 1.3.0 is out, I will update my server today
and see if it gets better / worse 😊
Aksel D.
BDO
________________________________
Fra: rsyslog <[email protected]> på vegne av Damiano Verzulli
<[email protected]>
Sendt: tirsdag 11. desember 2018 16.13.30
Til: [email protected]
Emne: Re: [rsyslog] CPU at 100% while using imrelp.
On 11/12/18 09:57, Aksel Cevdet Devrimci via rsyslog wrote:
[...]
System: Ubuntu 14.04(4.4 kernel) VM on KVM(8 cores and 16GB RAM),
rsyslog 3.39(ubuntu ppa) and librelp 1.2.18(ubuntu ppa).
^^^^^^^^^^^^
"rsyslog 3.39": is this correct ?
I'm asking as current rsyslog release is 8.40:
https://www.rsyslog.com/downloads/download-v8-stable/
So some typos occurred while writing your mail or either you're using such
an old rsyslog release to.... not get an answer, here, unless you'll update
your rsyslog version :-)
Cheers,
DV
--
Damiano Verzulli
e-mail: [email protected]
---
possible?ok:while(!possible){open_mindedness++}
---
"Technical people tend to fall into two categories: Specialists
and Generalists. The Specialist learns more and more about a
narrower and narrower field, until he eventually, in the limit,
knows everything about nothing. The Generalist learns less and
less about a wider and wider field, until eventually he knows
nothing about everything." - William Stucke - AfrISPA
http://elists.isoc.org/mailman/private/pubsoft/2007-December/001935.html
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
