And don't forget that impstats tells you exactly what is going on in rsyslog internally. Did you look at that?
Rainer Sent from phone, thus brief. Am 18.11.2016 15:39 schrieb "matthew.gaetano" <matthew.gaet...@gmx.ca>: > David Lang wrote > >> singh.janmejay wrote > >>> how many threads is imptcp configured for? > >> > >> On the reciever imptcp threads is set to 7, so 8 total (equal to the > >> number of cores on the server). > > > > how did you arrive at this number? Rsyslog can behave very badly with too > > many threads (the threads end up competing for locks on the main queue) > > Early on we were trying to troubleshoot load issues. We found the option > when reading the documentation and researching it further didn't result any > reasons not to use it, only that it should be set to no more than the > number > of course on the system. > > > David Lang wrote > > what is your batch size set to? > > 4096 on the sender side. > > > David Lang wrote > > You should only set the number of threads above 1 if you see that the > > worker threads are hitting max CPU, and try increasing batch size first > > before adding additional threads. > > > > you can see the per-thread usage on TOP by hitting 'H' > > Didnt know that; thank you. This mornings numbers clearly show there is no > justification for increasing it. going to wait until later in the day to > check again (higher load) and then adjust the threads accordingly. > > > > singh.janmejay wrote > > Sys% 55 and usr% 18 is very surprising. > > > > The other question I forgot to ask was around octate-counted vs > > octate-stuffed framing. But had that been a problem, it should have > > reflected as those percentages being the other way (usr% should have been > > significantly higher than sys%). > > > > If this is a new enough linux image, can you get debug symbols for Linux, > > libc and rsyslog and share output of "perf record -g -a -F 97" when it's > > healthy and when it's not? > > > > You'll need to launch "perf report" to view the data. If possible please > > share screenshots with all rows collapsed, and another one with the top > > few (upto >2%, say) expanded. > > > > Also, what was the load-avg at the time of degraded vs healthy receiver. > > > > Please do capture the netstat grep output if it happens again. > > > > Btw, it may be possible to reproduce it on-demand if its purely a > function > > of load. There is a utility called tcpflood in rsyslog tests directory > > that can flood messages quite well. Ruleset behind imptcp can be > > configured to filter and discard those messages. > > It seems we are not tracking that either at the moment; ill have to get it > added. Over the next few days Ill be looking at getting better answers to > the questions posed both for the healthy scenario and the faulty scenario. > In addition we will look into attempting to reproduce the issue using the > test method noted. > > I will get back to you as soon as i can. > > > > ----- > ~Regards > > Matthew Gaetano > -- > View this message in context: http://rsyslog-users.1305293. > n2.nabble.com/Disabling-IMPTCP-FlowControl-tp7591485p7591512.html > Sent from the rsyslog-users mailing list archive at Nabble.com. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
