On Thu, Jun 20, 2013 at 8:50 PM, David Lang <[email protected]> wrote: > what effect did this have on throughput and/or CPU load? > > I remember one case where processing rate got up from few thousand to several hundered thousand messages.
Rainer > David Lang > > On Thu, 20 Jun 2013, Boylan, James wrote: > > The configuration maxes out at 1000. At 1000 I was able to see the rate >> of closes drop for several thousand a minute to the hundreds. It was a >> significant change. >> >> - James >> >> >> -----Original Message----- >> From: >> [email protected].**com<[email protected]>[mailto: >> rsyslog-bounces@lists.**adiscon.com <[email protected]>] >> On Behalf Of David Lang >> Sent: Thursday, June 20, 2013 8:55 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] imPTCP module >> >> On Thu, 20 Jun 2013, Boylan, James wrote: >> >> I had recently noticed the error from the config of >>> $InputPTCPServerHelperThreads and had commented it out when I did notice >>> it. >>> Now that I know the correct option I've adjusted the configs >>> accordingly and it is running with the expected number of threads which >>> is good to see. >>> >>> That aside, the increased DynafileCacheSize has definitely had a >>> positive improvement overall. We definitely appreciate the input we've >>> gotten trying to implement the tuning options. >>> >> >> how big a difference did this make? >> >> David Lang >> >> We're going to be looking at testing out 7.4.1 soon (We're running on >>> 7.2.5 at the moment) to see what kind of performance gains can be seen in >>> the improvements between those versions. >>> >>> -- James >>> >>> >>> -----Original Message----- >>> From: [email protected].**com<[email protected]> >>> [mailto:rsyslog-bounces@lists.**adiscon.com<[email protected]>] >>> On Behalf Of Rainer >>> Gerhards >>> Sent: Thursday, June 20, 2013 4:01 AM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] imPTCP module >>> >>> On Tue, Jun 18, 2013 at 3:30 PM, David Lang <[email protected]> wrote: >>> >>> The overhead of the opens and closes is so high that I expect that >>>> you just need to scale it to the point where you are keeping them open. >>>> >>>> If it's set a lot larger than what you need it to be, it wastes >>>> memory that you could use for other things (I don't know how much) >>>> >>> >>> >>> It's depending on buffer parameters. By default I think two 64k buffers >>> (but I may be wrong). >>> >>> >>> , and I guess if it's too large it could be expensive to search and >>>> find that something isn't in there. >>>> >>>> >>> In current v7, that's no longer a problem, we have switched to a hash >>> table lookup. Seen some cases with low-thousands of open files and good >>> performance (that actually made us switch ;)). >>> >>> >>> >>>> But I would expect that these would be fairly minor effects. I don't >>>> understand why the default is so low. >>>> >>>> >>>> Stems back to pre-journald times, when we weighted SOHO vs. enterprise >>> use case. I should probably now go a bit higher. >>> >>> Rainer >>> >>> David Lang >>>> >>>> >>>> On Tue, 18 Jun 2013, Boylan, James wrote: >>>> >>>> We definitely do have many files being created. >>>> >>>>> >>>>> I'm starting to do the strace and I see what you mean about tons of >>>>> open and close actions. At what point does increasing >>>>> DynaFileCacheSize actually start negatively impacting overall >>>>> performance? Is there a number that we should keep the cache size >>>>> under? Or does it just need to be scaled based on the performance of >>>>> the hardware it is running on? >>>>> >>>>> -- James >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: [email protected].****com<rsyslog-bounces@lists.** >>>>> adiscon.com <[email protected]>>[mailto: >>>>> rsyslog-bounces@lists.**adisco**n.com <http://adiscon.com> >>>>> <rsyslog-bounces@lists.**adiscon.com<[email protected]> >>>>> >] >>>>> On Behalf Of David Lang >>>>> Sent: Monday, June 17, 2013 4:07 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] imPTCP module >>>>> >>>>> On Mon, 17 Jun 2013, Boylan, James wrote: >>>>> >>>>> Per David and Rainer's suggestion, I've cut us over to this module. >>>>> >>>>>> Definitely an improvement for performance. >>>>>> >>>>>> I do have one question. The configuration option >>>>>> $InputPTCPHelperThreads doesn't seem to do anything. I have it set >>>>>> to 12 (It's a 23 core machine) but it only ever creates 3 threads for >>>>>> the imptcp module. >>>>>> >>>>>> >>>>> I think it will use one thread per inbound connection, up to the max. >>>>> >>>>> If I remember your prior posts, you only had a handful of systems >>>>> sending you connections, but they were sending them at very high >>>>> rates (I could very easily be mixing you up with the other team that >>>>> had thousands of hosts sending >>>>> connections) >>>>> >>>>> But in any case, this shows that your bottleneck is not on the input >>>>> side (at least not with imptcp), it's on the output side where you >>>>> are using 8 threads, each using about 1/4 of a core. >>>>> >>>>> This makes me think that you have problems in your ruleset that we >>>>> should look at optimizing. >>>>> >>>>> Am I correct in remembering you as the one who started off with 480 >>>>> very complex if statements and we simplified it down to ~30 if >>>>> statements? >>>>> >>>>> If so, one thing that you need to do is to increase the number of >>>>> different files that it keeps track of. >>>>> >>>>> DynaFileCacheSize defaults to keeping track of 10 files. Since you >>>>> have >>>>> ~500 files that you are writing to, I think that you need to set >>>>> this to >>>>> 500 or higher. >>>>> >>>>> I'll bet that if you were to do a strace of those main Q threads you >>>>> would find that they are doing a lot of opening and closing of files >>>>> (pretty close to every message), and increasing the >>>>> DynaFileCacheSize to something large enough to avoid that would >>>>> result in a very sharp decrease in the CPU needed, and an even >>>>> larger increase in the rate of messages written. >>>>> >>>>> David Lang >>>>> >>>>> 26694 root 20 0 15.9g 7.9g 1480 S 26.8 16.8 3:44.63 rs:main >>>>> >>>>>> Q:Reg >>>>>> 26695 root 20 0 15.9g 7.9g 1480 R 26.3 16.8 3:44.89 rs:main >>>>>> Q:Reg >>>>>> 26689 root 20 0 15.9g 7.9g 1480 S 23.8 16.8 3:46.23 rs:main >>>>>> Q:Reg >>>>>> 26693 root 20 0 15.9g 7.9g 1480 S 23.5 16.8 3:45.76 rs:main >>>>>> Q:Reg >>>>>> 26698 root 20 0 15.9g 7.9g 1480 S 23.5 16.8 3:44.26 rs:main >>>>>> Q:Reg >>>>>> 26697 root 20 0 15.9g 7.9g 1480 S 22.8 16.8 3:43.07 rs:main >>>>>> Q:Reg >>>>>> 26699 root 20 0 15.9g 7.9g 1480 S 22.8 16.8 3:45.14 rs:main >>>>>> Q:Reg >>>>>> 26696 root 20 0 15.9g 7.9g 1480 S 22.0 16.8 3:46.56 rs:main >>>>>> Q:Reg >>>>>> 26685 root 20 0 15.9g 7.9g 1480 S 1.8 16.8 0:48.19 in:imptcp >>>>>> 26690 root 20 0 15.9g 7.9g 1480 S 1.8 16.8 0:28.76 in:imptcp >>>>>> 26692 root 20 0 15.9g 7.9g 1480 S 1.0 16.8 0:26.70 in:imptcp >>>>>> 26682 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 rsyslogd >>>>>> 26683 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:immark >>>>>> 26684 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:imudp >>>>>> 26686 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 >>>>>> in:imuxsock >>>>>> 26687 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:imklog >>>>>> 26688 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 >>>>>> in:impstats >>>>>> >>>>>> --James >>>>>> >>>>>> ______________________________****_________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>>> <http:**//lists.ad <http://lists.ad> >>>>>> i >>>>>> scon.net/mailman/listinfo/**rsyslog<http://scon.net/mailman/listinfo/rsyslog> >>>>>> > >>>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>>> <http://**www.rsyslog <http://www.rsyslog>. >>>>>> c om/professional-services/> What's up with rsyslog? Follow >>>>>> https://twitter.com/rgerhards NOTE >>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>> you DON'T LIKE THAT. >>>>>> >>>>>> ______________________________****_________________ >>>>>> >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>> <http:**//lists.adi <http://lists.adi> >>>>> s >>>>> con.net/mailman/listinfo/**rsyslog<http://con.net/mailman/listinfo/rsyslog> >>>>> > >>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>> <http://**www.rsyslog.c <http://www.rsyslog.c> >>>>> o m/professional-services/> What's up with rsyslog? Follow >>>>> https://twitter.com/rgerhards NOTE WELL: >>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> ______________________________****_________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>> <http:**//lists.adi <http://lists.adi> >>>>> s >>>>> con.net/mailman/listinfo/**rsyslog<http://con.net/mailman/listinfo/rsyslog> >>>>> > >>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>> <http://**www.rsyslog.c <http://www.rsyslog.c> >>>>> o m/professional-services/> What's up with rsyslog? Follow >>>>> https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing >>>>> list, posts are ARCHIVED by a myriad of sites beyond our control. >>>>> PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. >>>>> >>>>> ______________________________****_________________ >>>>> >>>> rsyslog mailing list >>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>> <http:**//lists.adis <http://lists.adis> >>>> c >>>> on.net/mailman/listinfo/**rsyslog<http://on.net/mailman/listinfo/rsyslog> >>>> > >>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>> <http://**www.rsyslog.co <http://www.rsyslog.co> >>>> m /professional-services/> What's up with rsyslog? Follow >>>> https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing >>>> list, posts are ARCHIVED by a myriad of sites beyond our control. >>>> PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. >>>> >>>> ______________________________**_________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE >>> THAT. >>> ______________________________**_________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >>> >>> ______________________________**_________________ >> rsyslog mailing list >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE >> THAT. >> ______________________________**_________________ >> rsyslog mailing list >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
