Re: [rsyslog] rulesets, threads, queues

Sat, 15 Jun 2013 14:17:15 -0700 

On Sat, 15 Jun 2013, Timothy Ehlers wrote:


We resolved our issue with this by spawning more main Q threads (8) and
removed the Action ques.

CPU usage for rsyslogd on the 50mb/sec box is sitting steadily at 200% (was
100).

We will try and tune the regex as well.
If tuning the regex doesn't help enough, one other thing that can be done is to convert the string generation to be a plugin module in C 

I believe that Adiscon can write this for you for around 500 euro
When we introduced the idea of C string generation, there was doubt if it would help noticably, IIRC we found that it gained 10%+ performance when dealing with the standard formats (which are very simple).
With something like your strings, where you are using a regex, but where I think the C code to make the strings you want would be very trivial, I think the gains would be huge
If I'm deciphering your intent correctly from the regex listed, what you are wanting is the first part of the program name, up until the first non-alphanumeric character.
I think that converting the string generation to be a plugin would probably reduce your CPU utilization from 200% to under 50%. 

David Lang


On Sat, Jun 15, 2013 at 5:09 AM, Rainer Gerhards
<[email protected]>wrote:


I have updated the blog post (and fixed the typo):

http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html

Rainer


On Sat, Jun 15, 2013 at 11:57 AM, Rainer Gerhards
<[email protected]>wrote:





On Sat, Jun 15, 2013 at 9:49 AM, David Lang <[email protected]> wrote:


In this particular case, there was one action in the ruleset (a rather
ugly dynafile with a regex in the filename and two in the format

string).

Based on this writeup it sounds as if multiple threads wouldn't help
(either on the mail or on the action queue)

assuming the same number of actions, I am not understanding why there
would be any difference between trying to use multiple threads on a main
queue or an action queue.



because more work happens on the main queue --> filtering, obtaining
batches and a couple of minor other things.

Rainer


by the way, I think you mix up sync and async at one point in the post

"The reason is that main queue workers process all filters for all rules
inside the rule set, as well as full action processing for all actions

that

are not run on an synchronous (action) queue"

shouldn't this be asynchronous (action) queue instead?


David Lang

On Sat, 15 Jun 2013, Rainer Gerhards wrote:

 I managed to do some quick writeup. It's far from perfect, but may be

useful:

http://blog.gerhards.net/2013/**06/rsyslog-performance-main-**
and-action.html<

http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html>



If you have suggestions/questions, we can always extend or add more.
Family
is now asking me to go off the computer ;)

Rainer


On Sat, Jun 15, 2013 at 10:00 AM, Rainer Gerhards
<[email protected]>**wrote:

 The effect on action queues is limited.  Main q workers are the

primary

tool. Explaination in blog later.

Sent from phone, thus brief.
Am 15.06.2013 09:46 schrieb "David Lang" <[email protected]>:

after he setup action queues it was only doing one thread per action


queue, even after the maxthreads for the actionqueue was set to 64.

David Lang

On Sat, 15 Jun 2013, Rainer Gerhards wrote:

 Action or main q threads?



Sent from phone, thus brief.
Am 15.06.2013 09:23 schrieb "David Lang" <[email protected]>:

 He had set the max threads to 64 and was still only seeing one

thread



working.

In a few of the later messages in the thread, his actionqueue

config

parameters are up near the top of the message.

David Lang

On Sat, 15 Jun 2013, Rainer Gerhards wrote:

 Requires a longer reply, need to lookup some things myself first -
that's

 why I didn't throw in comments.


But one very important thing: by default a queue has max 1 worker.
This
needs to be increased if multiple ones are desired.  I guess that
was
the
main point missing. If you create an async action queue, you also
get
one
more thread. Probably that's why it looks how it looks.  I don't
know
the
parameters out of my head.

Sent from phone, thus brief.
Am 15.06.2013 03:40 schrieb "David Lang" <[email protected]>:

 clarifying some questions that have come up in discussions around

 rulesets, thread, and queues


When you tie a ruleset to a specific input, does that still use

the

same
main queue? or will the logs from that input go to a different

main

queue
(so the workers working on that queue will have no locking
contention
with
workers working on logs from a different input), or do they get

put

on a
single main queue.

how do output threads and rulesets interact?

from what was reported, it looks as if everything goes to one
queue,
and
only if you set action queues per-ruleset will you end up with
per-ruleset
threads (in this case, actually per-actionqueue threads)

Is omfile able to make use of multiple threads if you have a

single

action
(in this case, using dynafiles)


note, the documentation for omfile there is a bad entry
http://www.rsyslog.com/doc/********omfile.html<

http://www.rsyslog.com/doc/******omfile.html>

<http://www.**rsyslog.com/doc/****omfile.**html<

http://www.rsyslog.com/doc/****omfile.html>



<http://www.**rsyslog.com/doc/****omfile.html<

http://rsyslog.com/doc/**omfile.html>

<http://www.**rsyslog.com/doc/**omfile.html<

http://www.rsyslog.com/doc/**omfile.html>






 <http://www.**rsyslog.com/doc/****omfile.html<

http://rsyslog.com/doc/**omfile.html>

<http://rsyslog.**com/doc/omfile.html<

http://rsyslog.com/doc/omfile.html>




<http://www.**rsyslog.com/doc/**omfile.html<

http://rsyslog.com/doc/omfile.html>

<http://www.**rsyslog.com/doc/omfile.html<

http://www.rsyslog.com/doc/omfile.html>











 * DynaFileCacheSize (not mandatory, default will be used)

Defines a template to be used for the output.


I think this is the wrong description and it would be good to say
what
the
default is.

David Lang
______________________________********_________________
rsyslog mailing list
http://lists.adiscon.net/********mailman/listinfo/rsyslog<

http://lists.adiscon.net/******mailman/listinfo/rsyslog>

<http**://lists.adiscon.net/******mailman/listinfo/rsyslog<

http://lists.adiscon.net/****mailman/listinfo/rsyslog>



<http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog<

http://lists.adiscon.net/**mailman/**listinfo/rsyslog>

<htt**p://lists.adiscon.net/****mailman/listinfo/rsyslog<

http://lists.adiscon.net/**mailman/listinfo/rsyslog>






 <http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog<

http://lists.adiscon.net/**mailman/**listinfo/rsyslog>

<htt**p://lists.adiscon.net/mailman/****listinfo/rsyslog<

http://lists.adiscon.net/mailman/**listinfo/rsyslog>



<htt**p://lists.adiscon.net/**mailman/**listinfo/rsyslog<

http://lists.adiscon.net/mailman/**listinfo/rsyslog>

<htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<

http://lists.adiscon.net/mailman/listinfo/rsyslog>










 http://www.rsyslog.com/********professional-services/<

http://www.rsyslog.com/******professional-services/>

<http://**www.rsyslog.com/******professional-services/<

http://www.rsyslog.com/****professional-services/>





<http://**www.rsyslog.com/******professional-services/<

http://www.rsyslog.com/****professional-services/>

<http://**www.rsyslog.com/****professional-services/<

http://www.rsyslog.com/**professional-services/>






 <http://**www.rsyslog.com/****professional-**services/<

http://www.rsyslog.com/**professional-**services/>

<http:**//www.rsyslog.com/**professional-**services/<

http://www.rsyslog.com/professional-**services/>



<http:**//www.rsyslog.com/****professional-services/<

http://www.rsyslog.com/**professional-services/>

<http://**www.rsyslog.com/professional-**services/<

http://www.rsyslog.com/professional-services/>










 What's up with rsyslog? Follow https://twitter.com/rgerhards


NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST

if

you
DON'T LIKE THAT.

 ______________________________******_________________

 rsyslog mailing list

http://lists.adiscon.net/******mailman/listinfo/rsyslog<

http://lists.adiscon.net/****mailman/listinfo/rsyslog>

<http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<

http://lists.adiscon.net/**mailman/listinfo/rsyslog>



<http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<

http://lists.adiscon.net/mailman/**listinfo/rsyslog>

<htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<

http://lists.adiscon.net/mailman/listinfo/rsyslog>






 http://www.rsyslog.com/******professional-services/<

http://www.rsyslog.com/****professional-services/>

<http://**www.rsyslog.com/****professional-services/<

http://www.rsyslog.com/**professional-services/>



<http://**www.rsyslog.com/**professional-**services/<

http://www.rsyslog.com/professional-**services/>

<http:**//www.rsyslog.com/**professional-services/<

http://www.rsyslog.com/professional-services/>






 What's up with rsyslog? Follow https://twitter.com/rgerhards

NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.

 ______________________________******_________________

 rsyslog mailing list

http://lists.adiscon.net/******mailman/listinfo/rsyslog<

http://lists.adiscon.net/****mailman/listinfo/rsyslog>

<http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<

http://lists.adiscon.net/**mailman/listinfo/rsyslog>



<http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<

http://lists.adiscon.net/mailman/**listinfo/rsyslog>

<htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<

http://lists.adiscon.net/mailman/listinfo/rsyslog>






 http://www.rsyslog.com/******professional-services/<

http://www.rsyslog.com/****professional-services/>

<http://**www.rsyslog.com/****professional-services/<

http://www.rsyslog.com/**professional-services/>



<http://**www.rsyslog.com/**professional-**services/<

http://www.rsyslog.com/professional-**services/>

<http:**//www.rsyslog.com/**professional-services/<

http://www.rsyslog.com/professional-services/>






 What's up with rsyslog? Follow https://twitter.com/rgerhards

NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.

 ______________________________****_________________


rsyslog mailing list
http://lists.adiscon.net/****mailman/listinfo/rsyslog<

http://lists.adiscon.net/**mailman/listinfo/rsyslog>

<http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<

http://lists.adiscon.net/mailman/listinfo/rsyslog>



http://www.rsyslog.com/****professional-services/<

http://www.rsyslog.com/**professional-services/>

<http://**www.rsyslog.com/professional-**services/<

http://www.rsyslog.com/professional-services/>



What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if

you

DON'T LIKE THAT.

 ______________________________****_________________


rsyslog mailing list
http://lists.adiscon.net/****mailman/listinfo/rsyslog<

http://lists.adiscon.net/**mailman/listinfo/rsyslog>

<http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<

http://lists.adiscon.net/mailman/listinfo/rsyslog>



http://www.rsyslog.com/****professional-services/<

http://www.rsyslog.com/**professional-services/>

<http://**www.rsyslog.com/professional-**services/<

http://www.rsyslog.com/professional-services/>



What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if

you

DON'T LIKE THAT.



 ______________________________**_________________

rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<

http://lists.adiscon.net/mailman/listinfo/rsyslog>

http://www.rsyslog.com/**professional-services/<

http://www.rsyslog.com/professional-services/>

What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

 ______________________________**_________________

rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<

http://lists.adiscon.net/mailman/listinfo/rsyslog>

http://www.rsyslog.com/**professional-services/<

http://www.rsyslog.com/professional-services/>

What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.





_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.







_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to