David Lang wrote >> singh.janmejay wrote >>> how many threads is imptcp configured for? >> >> On the reciever imptcp threads is set to 7, so 8 total (equal to the >> number of cores on the server). > > how did you arrive at this number? Rsyslog can behave very badly with too > many threads (the threads end up competing for locks on the main queue)
Early on we were trying to troubleshoot load issues. We found the option when reading the documentation and researching it further didn't result any reasons not to use it, only that it should be set to no more than the number of course on the system. David Lang wrote > what is your batch size set to? 4096 on the sender side. David Lang wrote > You should only set the number of threads above 1 if you see that the > worker threads are hitting max CPU, and try increasing batch size first > before adding additional threads. > > you can see the per-thread usage on TOP by hitting 'H' Didnt know that; thank you. This mornings numbers clearly show there is no justification for increasing it. going to wait until later in the day to check again (higher load) and then adjust the threads accordingly. singh.janmejay wrote > Sys% 55 and usr% 18 is very surprising. > > The other question I forgot to ask was around octate-counted vs > octate-stuffed framing. But had that been a problem, it should have > reflected as those percentages being the other way (usr% should have been > significantly higher than sys%). > > If this is a new enough linux image, can you get debug symbols for Linux, > libc and rsyslog and share output of "perf record -g -a -F 97" when it's > healthy and when it's not? > > You'll need to launch "perf report" to view the data. If possible please > share screenshots with all rows collapsed, and another one with the top > few (upto >2%, say) expanded. > > Also, what was the load-avg at the time of degraded vs healthy receiver. > > Please do capture the netstat grep output if it happens again. > > Btw, it may be possible to reproduce it on-demand if its purely a function > of load. There is a utility called tcpflood in rsyslog tests directory > that can flood messages quite well. Ruleset behind imptcp can be > configured to filter and discard those messages. It seems we are not tracking that either at the moment; ill have to get it added. Over the next few days Ill be looking at getting better answers to the questions posed both for the healthy scenario and the faulty scenario. In addition we will look into attempting to reproduce the issue using the test method noted. I will get back to you as soon as i can. ----- ~Regards Matthew Gaetano -- View this message in context: http://rsyslog-users.1305293.n2.nabble.com/Disabling-IMPTCP-FlowControl-tp7591485p7591512.html Sent from the rsyslog-users mailing list archive at Nabble.com. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
