On Thu, 26 Sep 2013, Rainer Gerhards wrote:
On Thu, Sep 26, 2013 at 11:29 AM, David Lang <[email protected]> wrote:
mmmhh.. have I overlooked something (may be, especially as I was busy the
past days)? My current understanding is that I recommended several times
to
drop the action queues, but that never happend. At least I have not seen
any pstats without them. This is why I am so persistive with this point. I
would like to see *the exact same config* with and without the action
queues. That config must of course include imudp running at realtime
priority.
If I have overlooked this, would you be able to point me to the correct
reply?
This topic is a mess to try and find things (and there was a lot of stuff
tested with just the throughput numbers reported, no pstats or top reports)
Yeah, that's my problem as well. I event tried via the mailing list
archive, but the threading is hard to follow there, too.
Robert, could you test the same config you tried earlier (2 main threads),
but with the action queues removed entirely?
he did have a test with 8 main queue threads and no action queue threads,
and it was unable to keep up with 250K logs/sec
Yup, I remember that one. With my whishful thinking I though yesterday
morning the same test was done with just 2 main threads, and things had
improved. That *would have* made sense to me (at least some). But then I
realized that the config was changed to include action queues -- so there
was no valid conclusion that could be drawn from these two tests. Because
it was a totally different scenario.
actually, as I understand it, the test went from
8 main queue workers, no action queues no stats provided, but falling far short
of 250K logs/sec
8 main queue workers + 8 action queue workers per action stats provided
about the same throughput
to
8 main queue workers + 1 action queue worker per action stats provided
better throughput, but still falling short
to
2 main queue workers + 1 action queue worker per action stats provided
succeded in keeping up with 250K logs/sec, but unable to sustain 300K logs/sec
The reason I was looking at trying to use rulesets was to be able to split
the test overhead between different threads (and have a
queue per ruleset rather than per action)
what is the best way to do this?
At least in theory, there is no such good way. The reason again simply is
that omfile is such a quick action (not counting IO), that the putting it
into an action queue creates probably 5 times overhead (even more if we
run
into contention situations).
Splitting of filter evaluation to several rulesets does no better than
running the main queue on multiple threads.
ahh, I thought that there were other ways to do this
well, there are several ways to do thing, but they'll all end up with
running the filter evaluation on multiple threads. In *theory* running
additional queues just increases the overhead, so doing that off a single
main queue *should* perform best (again, I don't say theory is matching
practice, but we need to have very solid test data in controlled
environments to draw this conclusion).
Actually, what I was envioning would have a subset of the filters evaluated in
each thread, but since all logs go to each thread I think this is probably a
difference that doesn't really matter.
a couple of interesting things to note in the top results.
the imudp thread is comeing close to maxing out a cpu
that *could* be look contention due to too much concurrency (that's way 2
vs. 8 main threads make some sense to me for an improvement - but only
"some", so it would need to be further evaluated iff it turns out to be
that case).
ok.
the main queue threads have been eating about the same CPU (per thread)
with or without action queues and with 2 or 8 main threads
Those CPU stats I have seen (not all, and not in consisten config settings)
showed quite a bit of system time, even the majority. That smells like
locking contention.
ahh, I wasn't looking expicitly for system vs user time, I'll keep an eye out
for that.
What I am trying to find now is a baseline from which, in a *controlled
environment* we can modify settings one way or the other, and check the
results. Once we have several different such test data sets, we can begin
to draw conclusions. Then, we can begin to improve parts of the code and/or
config. IMHO a very simple config will probably perform best, as long as
imudp runs at realtime prio. If not, we need to change code, as theory does
not match practice then (or even go further and change assumptions, but I
haven't seen anything in the past 4 to 5 years that points into that
direction).
agreed.
This is why I myself are also very interested in this topic - but so far I
got very few data points that I can actually work with. I unfortunately
don't have the time (maybe even the equipment) right now to setup a
dedicated test lab for this, so I am in need of Robert's help (I know if I
had the lab and a couple of days to play with it, I'd probably get results
very quickly, but...).
same here. I've got the time right now, but not access to the servers (what I've
got here at home is not really suitable for this level of testing)
David Lang
Rainer
David Lang
Again, this is *theory*. I would like to see some solid indication that
the
theory is wrong (see test to be carried out above). Even if it proves to
be
wrong, splitting off to rulesets would probably not make much sense.
Instead it would be better to try to find out where the theory is wrong,
and correct that.
But I really can't say more unless I see this with and without action
queue
behaviour (with otherwise *unchanged* config during the tests).
Sorry I have no better answer.
Rainer
David Lang
______________________________****_________________
rsyslog mailing list
http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
<http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
<http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
