On Tue, 11 Jun 2013, Roberto Giordani wrote:
Hello Rainer, unfortunately I didn't found any reduction of workload with 7.4 release. :-) I reach at least 400 action on server with 4 CPU and
since your imtcp threads are not using a lot of CPU, it doesn't look like your limitation is on the input side
If you have 400 actions defined, that could account for a lot of the time you are spending on things. can you post the ruleset so we can see if there are oppurtunities to optimize it further?
David Lang
$MainMsgQueueType FixedArray $MainMsgQueueSize 2000000 $MainMsgQueueWorkerThreads 10 $MainMsgQueueDequeueBatchSize 5000 $MainMsgQueueSaveOnShutdown on Output of rsyslog-pstats2013-06-11T09:34:52.384374+02:00 fsp01 rsyslogd-pstats: action 483: processed=0 failed=0 2013-06-11T09:34:52.384378+02:00 fsp01 rsyslogd-pstats: action 484: processed=0 failed=0 2013-06-11T09:34:52.384381+02:00 fsp01 rsyslogd-pstats: action 485: processed=0 failed=0 2013-06-11T09:34:52.384385+02:00 fsp01 rsyslogd-pstats: imtcp(10514): submitted=989503 2013-06-11T09:34:52.384389+02:00 fsp01 rsyslogd-pstats: imtcp(10515): submitted=0 2013-06-11T09:34:52.384393+02:00 fsp01 rsyslogd-pstats: imtcp(10516): submitted=55965412 2013-06-11T09:34:52.384399+02:00 fsp01 rsyslogd-pstats: main Q: size=1277408 enqueued=57252595 full=0 discarded.full=0 discarded.nf=0 maxqsize=1297183Attached threads running. Any help? Regarda, Roberto. On 06/06/2013 01:52 PM, Rainer Gerhards wrote:7.4.0 today Sent from phone, thus brief. Am 06.06.2013 13:18 schrieb "Roberto Giordani" <[email protected]>:Hello Rainer, do you have the release date for 7.13.15 stable? Regards, Roberto. On 06/06/2013 11:46 AM, Rainer Gerhards wrote:On Thu, Jun 6, 2013 at 11:41 AM, Roberto Giordani <[email protected]wrote:Hello Rainer,I'm looking to optimize the action.... Could you please convert my action as you describe ?I do not need to convert, you need to install the new version ;) I need to reproduce client log application to server and each log withthe same name but with specific owner. So I have about 30 files sent from the 20 clients and on the server for each client I need to check tag and create the output file with a specific owner. different for each log. Is there some "case statement" based on syslogtag? no, not yet ;)Rainer Please reply with your conversion of my action statementRegards, Roberto. On 06/05/2013 10:58 AM, Rainer Gerhards wrote: I think these directives cause the problem:if $fromhost-ip == "10.10.1.7" and $syslogfacility-text == "local6" and $syslogseverity-text == "debug" and $syslogtag == "TEST1" then action(type="omfile" DirOwner="user1" DirCreateMode="0750" FileCreateMode="0444" File="/rsyslog-data/file1.log" ) IIRC, there are many (if not all) versions of 7.2 which do exactly what you tell the, that is a) convert facility to a text b) do a string match on this text ... and do so for each of the properties. This is a very time consuming process. In 7.3.15+, the script optimizer greatly reduces that workload by detecting that what you really want to do is a very simple PRI-based filter ("prifilt(local6.=debug)"). While the latter requires roughly 10 CPU cycles, the former requires several ten-thousands.However, the work should be spread up on several CPUs, at least if thereare sufficiently large batches inside the system. This may not be the case here. Rainer On Wed, Jun 5, 2013 at 10:53 AM, Rainer Gerhards <[email protected]>****wrote: On Tue, Jun 4, 2013 at 5:07 AM, Eric <[email protected]> wrote:Unless you absolutely need TCP you'll gain some performance on switchingto UDP. Sorry, Eric, need to correct you here: TCP is much faster. A prime reasonis that for UDP, you need to do a system call for each messages. WithTCP, we usually receive several hundered to thousand with a single system call. RainerI have been pushing over 30k messages a second (UDP) with 1 input andtwooutputs. I've still not been able to make the boxes flinch (dell r420,GigE). I'm running a 7.2 variant on cent 6.2 with no real major performance tuning. Eric On Jun 3, 2013, at 2:48 PM, Roberto Giordani <[email protected]> wrote: Hello, I'm working on a project where 20 servers RHEL 5.8 (with rsyslog 5.8.12) has 20 input files on input and send about 10.000 messages for second to one rsyslog server 7.2 version The network is gigabit between client-->server and this is the daily nmon network traffic graphs The first configuration was with queue file, but is was too slow, so I've used the LinkedList queue. The current client settings are $InputFileName /file1.log $InputFileTag TEST1 $InputFileStateFile file1 $InputFileSeverity debug $InputFileFacility local6 $InputRunFileMonitor $InputFilePersistStateInterval 10 .... .... .... ##############################****############## $MaxMessageSize 9000 $MainMsgQueueType LinkedList $MainMsgQueueSize 1000000 $MainMsgQueueWorkerThreads 20 $MainMsgQueueDequeueBatchSize 5000 $MainMsgQueueSaveOnShutdown on $ActionQueueType LinkedList $ActionQueueSize 2000000 $ActionQueueWorkerThreads 50 $ActionQueueDequeueBatchSize 5000 $ActionQueueSaveOnShutdown on ##############################****############# $ActionResumeRetryCount -1 $ActionQueueTimeoutEnqueue 1 $****ActionSendResendLastMsgOnRecon****nect on $ActionQueueCheckpointInterval 1 local6.debug @@10.10.1.10:10514 The server settings are ##############################****################ # Provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 10514 #### GLOBAL DIRECTIVES #### # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit $ActionFileEnableSync off $umask 0007 if $fromhost-ip == "10.10.1.7" and $syslogfacility-text == "local6" and $syslogseverity-text == "debug" and $syslogtag == "TEST1" then action(type="omfile" DirOwner="user1" DirCreateMode="0750" FileCreateMode="0444" File="/rsyslog-data/file1.log"****) if $fromhost-ip == ........ ##############################****# My questions are: 1)how to find the right combination of Main queue ---> Action Queue---> receiver queue on rsyslog server ? 2)Is it possible to increase incoming messages and handle quickly? 3)Why the output log on Centralized Rsyslog are still behind the source log on the client? I've activated the pstats module to understand the queue status on clients and server. After 6 hours running the client has this report ......... 2013-06-03T22:23:16.708288+02:****00 app01 rsyslogd-pstats: action 9 queue: size=2000000 enqueued=9327675 full=6831507 maxqsize=20 00000 2013-06-03T22:23:16.708297+02:****00 app01 rsyslogd-pstats: main Q: size=2 enqueued=9648448 full=0 maxqsize=20395 2013-06-03T22:23:46.708367+02:****00 app01 rsyslogd-pstats: imuxsock: submitted=323414 ratelimit.discarded=0 ratelimit.numratelimi ters=29974 2013-06-03T22:23:46.708382+02:****00 app01 rsyslogd-pstats: action 9 queue: size=2000000 enqueued=9340578 full=6844410 maxqsize=20 00000 2013-06-03T22:23:46.708390+02:****00 app01 rsyslogd-pstats: main Q: size=2164 enqueued=9666464 full=0 maxqsize=20395 2013-06-03T22:24:16.708923+02:****00 app01 rsyslogd-pstats: imuxsock: submitted=328198 ratelimit.discarded=0 ratelimit.numratelimi ters=29986 2013-06-03T22:24:16.708941+02:****00 app01 rsyslogd-pstats: action 9 queue: size=2000000 enqueued=9355649 full=6859481 maxqsize=20 00000 2013-06-03T22:24:16.708949+02:****00 app01 rsyslogd-pstats: main Q: size=1364 enqueued=9686593 full=0 maxqsize=20395 2013-06-03T22:24:46.709300+02:****00 app01 rsyslogd-pstats: imuxsock: submitted=333070 ratelimit.discarded=0 ratelimit.numratelimi ters=29997 2013-06-03T22:24:46.709316+02:****00 app01 rsyslogd-pstats: action 9 queue: size=2000000 enqueued=9365276 full=6869108 maxqsize=20 00000 2013-06-03T22:24:46.709323+02:****00 app01 rsyslogd-pstats: main Q: size=2123 enqueued=9702047 full=0 maxqsize=20395 2013-06-03T22:25:16.709807+02:****00 app01 rsyslogd-pstats: imuxsock: submitted=337951 ratelimit.discarded=0 ratelimit.numratelimi ters=30009 2013-06-03T22:25:16.709823+02:****00 app01 rsyslogd-pstats: action 9 queue: size=2000000 enqueued=9379492 full=6883324 maxqsize=20 00000 2013-06-03T22:25:16.709832+02:****00 app01 rsyslogd-pstats: main Q: size=2 enqueued=9719723 full=0 maxqsize=20395 2013-06-03T22:25:46.709942+02:****00 app01 rsyslogd-pstats: imuxsock: submitted=343014 ratelimit.discarded=0 ratelimit.numratelimi ters=30021 2013-06-03T22:25:46.709980+02:****00 app01 rsyslogd-pstats: action 9 queue: size=2000000 enqueued=9389640 full=6893472 maxqsize=20 00000 and the rsyslog server the following stats ....... 2013-06-03T23:05:15.898682+02:****00 fsp01 rsyslogd-pstats: main Q: size=9978 enqueued=5032165 full=156941 discarded.full=0 discarded.nf=0 maxqsize=10000 2013-06-03T23:06:56.157199+02:****00 fsp01 rsyslogd-pstats: main Q: size=9987 enqueued=5065134 full=157971 discarded.full=0 discarded.nf=0 maxqsize=10000 2013-06-03T23:08:30.657673+02:****00 fsp01 rsyslogd-pstats: main Q: size=9972 enqueued=5096315 full=158942 discarded.full=0 discarded.nf=0 maxqsize=10000 2013-06-03T23:10:09.895850+02:****00 fsp01 rsyslogd-pstats: main Q: size=9986 enqueued=5129162 full=159969 discarded.full=0 discarded.nf=0 maxqsize=10000 2013-06-03T23:11:42.488505+02:****00 fsp01 rsyslogd-pstats: main Q: size=9973 enqueued=5159935 full=160933 discarded.full=0 discarded.nf=0 maxqsize=10000 2013-06-03T23:13:23.213800+02:****00 fsp01 rsyslogd-pstats: main Q: size=9973 enqueued=5193246 full=161973 discarded.full=0 discarded.nf=0 maxqsize=10000 2013-06-03T23:14:58.833570+02:****00 fsp01 rsyslogd-pstats: main Q: size=9970 enqueued=5224922 full=162962 discarded.full=0 discarded.nf=0 maxqsize=10000 2013-06-03T23:16:35.184133+02:****00 fsp01 rsyslogd-pstats: main Q: size=9975 enqueued=5256863 full=163960 discarded.full=0 discarded.nf=0 maxqsize=10000 2013-06-03T23:18:13.992958+02:****00 fsp01 rsyslogd-pstats: main Q: size=9991 enqueued=5289392 full=164977 discarded.full=0 discarded.nf=0 maxqsize=10000 2013-06-03T23:19:52.464473+02:****00 fsp01 rsyslogd-pstats: main Q: size=9942 enqueued=5322013 full=165996 discarded.full=0 discarded.nf=0 maxqsize=10000 4)Why on the client enqueued value never decrease and full= is always different of 0? I'm planning to distribute the 20 client to 2 process of rsyslog server on different port on the same server but I think your help about the right combination of action queue on client and main queue on server. On rsyslog documentation I've found a lot of info about to handle inputqueue (# thread, max messages and DequeuBatch) but I dont' know how torsyslog should better work on receiver server.I've seen that the rsyslog 7.2 process on server use only one CPU eachtime, while the server has 4CPU and 12GB ram on 64bit O.S. 5)Some option during configuration process can change this behavior ? $ ./configure --prefix=/usr/local/rsyslog7 --enable-gnutls --enable-imfile --enable-omruleset --enable-imptcp --enable-mmnormalize --enable-usertools --enable-imdiag --enable-diagtools --enable-impstats Regards, Roberto. ______________________________****_________________ rsyslog mailing list http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriadof sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if youDON'T LIKE THAT. ______________________________****_________________ rsyslog mailing list http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriadof sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if youDON'T LIKE THAT.______________________________****_________________rsyslog mailing list http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.______________________________**_________________ rsyslog mailing list http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
<<ATTACHMENT: workload_7.4.png>>
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
