The overhead of the opens and closes is so high that I expect that you just need
to scale it to the point where you are keeping them open.
If it's set a lot larger than what you need it to be, it wastes memory that you
could use for other things (I don't know how much), and I guess if it's too
large it could be expensive to search and find that something isn't in there.
But I would expect that these would be fairly minor effects. I don't understand
why the default is so low.
David Lang
On Tue, 18 Jun 2013, Boylan, James wrote:
We definitely do have many files being created.
I'm starting to do the strace and I see what you mean about tons of open and
close actions. At what point does increasing DynaFileCacheSize actually start
negatively impacting overall performance? Is there a number that we should keep
the cache size under? Or does it just need to be scaled based on the
performance of the hardware it is running on?
-- James
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of David Lang
Sent: Monday, June 17, 2013 4:07 PM
To: rsyslog-users
Subject: Re: [rsyslog] imPTCP module
On Mon, 17 Jun 2013, Boylan, James wrote:
Per David and Rainer's suggestion, I've cut us over to this module. Definitely
an improvement for performance.
I do have one question. The configuration option $InputPTCPHelperThreads
doesn't seem to do anything. I have it set to 12 (It's a 23 core machine) but
it only ever creates 3 threads for the imptcp module.
I think it will use one thread per inbound connection, up to the max.
If I remember your prior posts, you only had a handful of systems sending you
connections, but they were sending them at very high rates (I could very easily
be mixing you up with the other team that had thousands of hosts sending
connections)
But in any case, this shows that your bottleneck is not on the input side (at
least not with imptcp), it's on the output side where you are using 8 threads,
each using about 1/4 of a core.
This makes me think that you have problems in your ruleset that we should look
at optimizing.
Am I correct in remembering you as the one who started off with 480 very
complex if statements and we simplified it down to ~30 if statements?
If so, one thing that you need to do is to increase the number of different
files that it keeps track of.
DynaFileCacheSize defaults to keeping track of 10 files. Since you have ~500
files that you are writing to, I think that you need to set this to 500 or
higher.
I'll bet that if you were to do a strace of those main Q threads you would find
that they are doing a lot of opening and closing of files (pretty close to
every message), and increasing the DynaFileCacheSize to something large enough
to avoid that would result in a very sharp decrease in the CPU needed, and an
even larger increase in the rate of messages written.
David Lang
26694 root 20 0 15.9g 7.9g 1480 S 26.8 16.8 3:44.63 rs:main Q:Reg
26695 root 20 0 15.9g 7.9g 1480 R 26.3 16.8 3:44.89 rs:main Q:Reg
26689 root 20 0 15.9g 7.9g 1480 S 23.8 16.8 3:46.23 rs:main Q:Reg
26693 root 20 0 15.9g 7.9g 1480 S 23.5 16.8 3:45.76 rs:main Q:Reg
26698 root 20 0 15.9g 7.9g 1480 S 23.5 16.8 3:44.26 rs:main Q:Reg
26697 root 20 0 15.9g 7.9g 1480 S 22.8 16.8 3:43.07 rs:main Q:Reg
26699 root 20 0 15.9g 7.9g 1480 S 22.8 16.8 3:45.14 rs:main Q:Reg
26696 root 20 0 15.9g 7.9g 1480 S 22.0 16.8 3:46.56 rs:main Q:Reg
26685 root 20 0 15.9g 7.9g 1480 S 1.8 16.8 0:48.19 in:imptcp
26690 root 20 0 15.9g 7.9g 1480 S 1.8 16.8 0:28.76 in:imptcp
26692 root 20 0 15.9g 7.9g 1480 S 1.0 16.8 0:26.70 in:imptcp
26682 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 rsyslogd
26683 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:immark
26684 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:imudp
26686 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:imuxsock
26687 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:imklog
26688 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:impstats
--James
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
