Turned off TLS and it dropped to 10% ish so looks like gnutls might be the problem. AES-NI is active on the VM(tested with openssl), but not sure how to test with gnutls.
Tried to set the env variable GNUTLS_CPUID_OVERRIDE(https://www.gnutls.org/manual/gnutls.html#gnutls_005fglobal_005finit) to both off and on but there where no difference. Same goes when we disable AES-NI support on the VM. Still stuck at 5,8K while using relp with gnutls. Tried with some different options with the tls.prioritystring but had no effect. Aksel D. BDO ________________________________ Fra: Simon Lundström <[email protected]> Sendt: onsdag 12. desember 2018 13.59.00 Til: Aksel Cevdet Devrimci Kopi: [email protected] Emne: Re: [rsyslog] CPU at 100% while using imrelp. On Wed, 2018-12-12 at 12:30:05 +0100, Aksel Cevdet Devrimci wrote: >As you can see it's the imrelp thread that is always at a 100% And you said so in your first mail, I missed that sorry. Can you push more than 5.8k EPS through rsyslog even though it's using 100% CPU? >Upgrading librelp and rsyslog to the latest version did not have any effect. > > >Is it possible to have multiple imrelp threads? Doesn't look like it, someone from Adiscon will have to answer that though. Can you, as a test, turn off TLS to see if it's the decryption which uses CPU? If not this might be a bug in RELP? If it's the decryption which uses a lot of CPU: * Is it a VM? Does the VM have access to the HW crypto accelerator? Is GnuTLS using the crypto acc? (I don't know how to check this). * Are the clients doing a lot of short sessions? Can you make sure they are using very long connection sessions? RSA, which I'm guessing is the default (from <https://github.com/rsyslog/librelp/blob/bd6c0a7fc7dd2dbedf772c1de8a500ad602a8c97/src/tcp.c#L1144> and <https://gnutls.org/manual/html_node/Priority-Strings.html>), is CPU intensive so long living sessions are prefered. * ECDSA should be faster than RSA, you can set it via <https://www.rsyslog.com/doc/v8-stable/configuration/modules/imrelp.html#tls-prioritystring> and <https://gnutls.org/manual/html_node/Priority-Strings.html> Basically everything that applies to making HTTPS fast applies here as well. Good luck and please report back of your, hopefully, great success = ) BR, - Simon >________________________________ >Fra: Simon Lundström <[email protected]> >Sendt: onsdag 12. desember 2018 10.53.59 >Til: rsyslog-users >Kopi: Aksel Cevdet Devrimci >Emne: Re: [rsyslog] CPU at 100% while using imrelp. > >Be sure to check what is using CPU too. If you're using Linux and GNU >top you can use `top -H -p $(pgrep rsyslog)` to see the threads (or >press H after you've started top IIRC). > >Each module has one or multiple threaded workers so you can easily see >what is using CPU. > >BR, >- Simon > >On Wed, 2018-12-12 at 09:36:18 +0100, Aksel Cevdet Devrimci via rsyslog wrote: >>No my bad, is ofc 8.39 :) >> >> >>Now that rsyslog 8.40 and librelp 1.3.0 is out, I will update my server today >>and see if it gets better / worse 😊 >> >> >>Aksel D. >> >>BDO >> >>________________________________ >>Fra: rsyslog <[email protected]> på vegne av Damiano Verzulli >><[email protected]> >>Sendt: tirsdag 11. desember 2018 16.13.30 >>Til: [email protected] >>Emne: Re: [rsyslog] CPU at 100% while using imrelp. >> >>On 11/12/18 09:57, Aksel Cevdet Devrimci via rsyslog wrote: >>> [...] >>> System: Ubuntu 14.04(4.4 kernel) VM on KVM(8 cores and 16GB RAM), >>> rsyslog 3.39(ubuntu ppa) and librelp 1.2.18(ubuntu ppa). >>> ^^^^^^^^^^^^ >> >>"rsyslog 3.39": is this correct ? >> >>I'm asking as current rsyslog release is 8.40: >> https://www.rsyslog.com/downloads/download-v8-stable/ >> >>So some typos occurred while writing your mail or either you're using such >>an old rsyslog release to.... not get an answer, here, unless you'll update >>your rsyslog version :-) >> >>Cheers, >>DV >> >>-- >>Damiano Verzulli >>e-mail: [email protected] >>--- >>possible?ok:while(!possible){open_mindedness++} >>--- >>"Technical people tend to fall into two categories: Specialists >>and Generalists. The Specialist learns more and more about a >>narrower and narrower field, until he eventually, in the limit, >>knows everything about nothing. The Generalist learns less and >>less about a wider and wider field, until eventually he knows >>nothing about everything." - William Stucke - AfrISPA >> http://elists.isoc.org/mailman/private/pubsoft/2007-December/001935.html >> >> >>_______________________________________________ >>rsyslog mailing list >>http://lists.adiscon.net/mailman/listinfo/rsyslog >>http://www.rsyslog.com/professional-services/ >>What's up with rsyslog? Follow https://twitter.com/rgerhards >>NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
