On Thu, Jun 20, 2013 at 5:18 PM, Boylan, James <[email protected]>wrote:
> The configuration maxes out at 1000. At 1000 I was able to see the rate of > closes drop for several thousand a minute to the hundreds. It was a > significant change. > > Just FYI: I thinjk I added cache-related pstats counters, which makes it much easier to find the best sizes. Rainer > - James > > > -----Original Message----- > From: [email protected] [mailto: > [email protected]] On Behalf Of David Lang > Sent: Thursday, June 20, 2013 8:55 AM > To: rsyslog-users > Subject: Re: [rsyslog] imPTCP module > > On Thu, 20 Jun 2013, Boylan, James wrote: > > > I had recently noticed the error from the config of > > $InputPTCPServerHelperThreads and had commented it out when I did notice > it. > > Now that I know the correct option I've adjusted the configs > > accordingly and it is running with the expected number of threads which > is good to see. > > > > That aside, the increased DynafileCacheSize has definitely had a > > positive improvement overall. We definitely appreciate the input we've > > gotten trying to implement the tuning options. > > how big a difference did this make? > > David Lang > > > We're going to be looking at testing out 7.4.1 soon (We're running on > 7.2.5 at the moment) to see what kind of performance gains can be seen in > the improvements between those versions. > > > > -- James > > > > > > -----Original Message----- > > From: [email protected] > > [mailto:[email protected]] On Behalf Of Rainer > > Gerhards > > Sent: Thursday, June 20, 2013 4:01 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] imPTCP module > > > > On Tue, Jun 18, 2013 at 3:30 PM, David Lang <[email protected]> wrote: > > > >> The overhead of the opens and closes is so high that I expect that > >> you just need to scale it to the point where you are keeping them open. > >> > >> If it's set a lot larger than what you need it to be, it wastes > >> memory that you could use for other things (I don't know how much) > > > > > > It's depending on buffer parameters. By default I think two 64k buffers > (but I may be wrong). > > > > > >> , and I guess if it's too large it could be expensive to search and > >> find that something isn't in there. > >> > > > > In current v7, that's no longer a problem, we have switched to a hash > table lookup. Seen some cases with low-thousands of open files and good > performance (that actually made us switch ;)). > > > > > >> > >> But I would expect that these would be fairly minor effects. I don't > >> understand why the default is so low. > >> > >> > > Stems back to pre-journald times, when we weighted SOHO vs. enterprise > use case. I should probably now go a bit higher. > > > > Rainer > > > >> David Lang > >> > >> > >> On Tue, 18 Jun 2013, Boylan, James wrote: > >> > >> We definitely do have many files being created. > >>> > >>> I'm starting to do the strace and I see what you mean about tons of > >>> open and close actions. At what point does increasing > >>> DynaFileCacheSize actually start negatively impacting overall > >>> performance? Is there a number that we should keep the cache size > >>> under? Or does it just need to be scaled based on the performance of > the hardware it is running on? > >>> > >>> -- James > >>> > >>> > >>> -----Original Message----- > >>> From: [email protected].**com< > [email protected]>[mailto: > >>> rsyslog-bounces@lists.**adiscon.com > >>> <[email protected]>] > >>> On Behalf Of David Lang > >>> Sent: Monday, June 17, 2013 4:07 PM > >>> To: rsyslog-users > >>> Subject: Re: [rsyslog] imPTCP module > >>> > >>> On Mon, 17 Jun 2013, Boylan, James wrote: > >>> > >>> Per David and Rainer's suggestion, I've cut us over to this module. > >>>> Definitely an improvement for performance. > >>>> > >>>> I do have one question. The configuration option > >>>> $InputPTCPHelperThreads doesn't seem to do anything. I have it set > >>>> to 12 (It's a 23 core machine) but it only ever creates 3 threads for > the imptcp module. > >>>> > >>> > >>> I think it will use one thread per inbound connection, up to the max. > >>> > >>> If I remember your prior posts, you only had a handful of systems > >>> sending you connections, but they were sending them at very high > >>> rates (I could very easily be mixing you up with the other team that > >>> had thousands of hosts sending > >>> connections) > >>> > >>> But in any case, this shows that your bottleneck is not on the input > >>> side (at least not with imptcp), it's on the output side where you > >>> are using 8 threads, each using about 1/4 of a core. > >>> > >>> This makes me think that you have problems in your ruleset that we > >>> should look at optimizing. > >>> > >>> Am I correct in remembering you as the one who started off with 480 > >>> very complex if statements and we simplified it down to ~30 if > statements? > >>> > >>> If so, one thing that you need to do is to increase the number of > >>> different files that it keeps track of. > >>> > >>> DynaFileCacheSize defaults to keeping track of 10 files. Since you > >>> have > >>> ~500 files that you are writing to, I think that you need to set > >>> this to > >>> 500 or higher. > >>> > >>> I'll bet that if you were to do a strace of those main Q threads you > >>> would find that they are doing a lot of opening and closing of files > >>> (pretty close to every message), and increasing the > >>> DynaFileCacheSize to something large enough to avoid that would > >>> result in a very sharp decrease in the CPU needed, and an even > >>> larger increase in the rate of messages written. > >>> > >>> David Lang > >>> > >>> 26694 root 20 0 15.9g 7.9g 1480 S 26.8 16.8 3:44.63 rs:main > >>>> Q:Reg > >>>> 26695 root 20 0 15.9g 7.9g 1480 R 26.3 16.8 3:44.89 rs:main > >>>> Q:Reg > >>>> 26689 root 20 0 15.9g 7.9g 1480 S 23.8 16.8 3:46.23 rs:main > >>>> Q:Reg > >>>> 26693 root 20 0 15.9g 7.9g 1480 S 23.5 16.8 3:45.76 rs:main > >>>> Q:Reg > >>>> 26698 root 20 0 15.9g 7.9g 1480 S 23.5 16.8 3:44.26 rs:main > >>>> Q:Reg > >>>> 26697 root 20 0 15.9g 7.9g 1480 S 22.8 16.8 3:43.07 rs:main > >>>> Q:Reg > >>>> 26699 root 20 0 15.9g 7.9g 1480 S 22.8 16.8 3:45.14 rs:main > >>>> Q:Reg > >>>> 26696 root 20 0 15.9g 7.9g 1480 S 22.0 16.8 3:46.56 rs:main > >>>> Q:Reg > >>>> 26685 root 20 0 15.9g 7.9g 1480 S 1.8 16.8 0:48.19 in:imptcp > >>>> 26690 root 20 0 15.9g 7.9g 1480 S 1.8 16.8 0:28.76 in:imptcp > >>>> 26692 root 20 0 15.9g 7.9g 1480 S 1.0 16.8 0:26.70 in:imptcp > >>>> 26682 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 rsyslogd > >>>> 26683 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:immark > >>>> 26684 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:imudp > >>>> 26686 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 > in:imuxsock > >>>> 26687 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:imklog > >>>> 26688 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 > in:impstats > >>>> > >>>> --James > >>>> > >>>> ______________________________**_________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.ad > >>>> i > >>>> scon.net/mailman/listinfo/rsyslog> > >>>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog. > >>>> c om/professional-services/> What's up with rsyslog? Follow > >>>> https://twitter.com/rgerhards NOTE > >>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>>> you DON'T LIKE THAT. > >>>> > >>>> ______________________________**_________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adi > >>> s > >>> con.net/mailman/listinfo/rsyslog> > >>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.c > >>> o m/professional-services/> What's up with rsyslog? Follow > >>> https://twitter.com/rgerhards NOTE WELL: > >>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> ______________________________**_________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adi > >>> s > >>> con.net/mailman/listinfo/rsyslog> > >>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.c > >>> o m/professional-services/> What's up with rsyslog? Follow > >>> https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing > >>> list, posts are ARCHIVED by a myriad of sites beyond our control. > >>> PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > >>> > >>> ______________________________**_________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adis > >> c > >> on.net/mailman/listinfo/rsyslog> > >> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.co > >> m /professional-services/> What's up with rsyslog? Follow > >> https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing > >> list, posts are ARCHIVED by a myriad of sites beyond our control. > >> PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE > THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE > THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
