7.4.0 today Sent from phone, thus brief. Am 06.06.2013 13:18 schrieb "Roberto Giordani" <[email protected]>:
> Hello Rainer, > do you have the release date for 7.13.15 stable? > > Regards, > Roberto. > > On 06/06/2013 11:46 AM, Rainer Gerhards wrote: > >> On Thu, Jun 6, 2013 at 11:41 AM, Roberto Giordani <[email protected] >> >wrote: >> >> Hello Rainer, >>> I'm looking to optimize the action.... >>> Could you please convert my action as you describe ? >>> >>> >> I do not need to convert, you need to install the new version ;) >> >> >> I need to reproduce client log application to server and each log with >>> the >>> same name but with specific owner. >>> So I have about 30 files sent from the 20 clients and on the server for >>> each client I need to check tag and create the output file with a >>> specific >>> owner. different for each log. >>> Is there some "case statement" based on syslogtag? >>> >>> >>> no, not yet ;) >> >> Rainer >> >> Please reply with your conversion of my action statement >>> >>> Regards, >>> Roberto. >>> >>> >>> On 06/05/2013 10:58 AM, Rainer Gerhards wrote: >>> >>> I think these directives cause the problem: >>>> >>>> >>>> if $fromhost-ip == "10.10.1.7" and $syslogfacility-text == "local6" and >>>> $syslogseverity-text == "debug" and $syslogtag == "TEST1" then >>>> action(type="omfile" DirOwner="user1" DirCreateMode="0750" >>>> FileCreateMode="0444" File="/rsyslog-data/file1.log" >>>> ) >>>> >>>> IIRC, there are many (if not all) versions of 7.2 which do exactly what >>>> you >>>> tell the, that is >>>> >>>> a) convert facility to a text >>>> b) do a string match on this text >>>> ... and do so for each of the properties. >>>> >>>> This is a very time consuming process. In 7.3.15+, the script optimizer >>>> greatly reduces that workload by detecting that what you really want to >>>> do >>>> is a very simple PRI-based filter ("prifilt(local6.=debug)"). While the >>>> latter requires roughly 10 CPU cycles, the former requires several >>>> ten-thousands. >>>> >>>> However, the work should be spread up on several CPUs, at least if there >>>> are sufficiently large batches inside the system. This may not be the >>>> case >>>> here. >>>> >>>> Rainer >>>> >>>> >>>> >>>> On Wed, Jun 5, 2013 at 10:53 AM, Rainer Gerhards >>>> <[email protected]>****wrote: >>>> >>>> >>>> On Tue, Jun 4, 2013 at 5:07 AM, Eric <[email protected]> wrote: >>>>> >>>>> Unless you absolutely need TCP you'll gain some performance on >>>>> switching >>>>> >>>>>> to UDP. >>>>>> >>>>>> Sorry, Eric, need to correct you here: TCP is much faster. A prime >>>>>> reason >>>>>> >>>>>> is that for UDP, you need to do a system call for each messages. With >>>>> TCP, >>>>> we usually receive several hundered to thousand with a single system >>>>> call. >>>>> >>>>> Rainer >>>>> >>>>> I have been pushing over 30k messages a second (UDP) with 1 input and >>>>> >>>>>> two >>>>>> outputs. I've still not been able to make the boxes flinch (dell r420, >>>>>> GigE). I'm running a 7.2 variant on cent 6.2 with no real major >>>>>> performance >>>>>> tuning. >>>>>> >>>>>> Eric >>>>>> >>>>>> On Jun 3, 2013, at 2:48 PM, Roberto Giordani <[email protected]> >>>>>> wrote: >>>>>> >>>>>> Hello, >>>>>> I'm working on a project where 20 servers RHEL 5.8 (with rsyslog >>>>>> 5.8.12) >>>>>> has 20 input files on input and send about 10.000 messages for second >>>>>> to >>>>>> one rsyslog server 7.2 version >>>>>> >>>>>> The network is gigabit between client-->server and this is the daily >>>>>> nmon >>>>>> network traffic graphs >>>>>> >>>>>> >>>>>> The first configuration was with queue file, but is was too slow, so >>>>>> I've >>>>>> used the LinkedList queue. >>>>>> >>>>>> The current client settings are >>>>>> $InputFileName /file1.log >>>>>> $InputFileTag TEST1 >>>>>> $InputFileStateFile file1 >>>>>> $InputFileSeverity debug >>>>>> $InputFileFacility local6 >>>>>> $InputRunFileMonitor >>>>>> $InputFilePersistStateInterval 10 >>>>>> .... >>>>>> .... >>>>>> .... >>>>>> ##############################****############## >>>>>> $MaxMessageSize 9000 >>>>>> >>>>>> $MainMsgQueueType LinkedList >>>>>> $MainMsgQueueSize 1000000 >>>>>> $MainMsgQueueWorkerThreads 20 >>>>>> $MainMsgQueueDequeueBatchSize 5000 >>>>>> $MainMsgQueueSaveOnShutdown on >>>>>> >>>>>> $ActionQueueType LinkedList >>>>>> $ActionQueueSize 2000000 >>>>>> $ActionQueueWorkerThreads 50 >>>>>> $ActionQueueDequeueBatchSize 5000 >>>>>> $ActionQueueSaveOnShutdown on >>>>>> ##############################****############# >>>>>> >>>>>> $ActionResumeRetryCount -1 >>>>>> $ActionQueueTimeoutEnqueue 1 >>>>>> $****ActionSendResendLastMsgOnRecon****nect on >>>>>> $ActionQueueCheckpointInterval 1 >>>>>> >>>>>> local6.debug @@10.10.1.10:10514 >>>>>> >>>>>> The server settings are >>>>>> ##############################****################ >>>>>> # Provides TCP syslog reception >>>>>> $ModLoad imtcp >>>>>> $InputTCPServerRun 10514 >>>>>> >>>>>> >>>>>> #### GLOBAL DIRECTIVES #### >>>>>> >>>>>> # Use default timestamp format >>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >>>>>> >>>>>> # File syncing capability is disabled by default. This feature is >>>>>> usually >>>>>> not required, >>>>>> # not useful and an extreme performance hit >>>>>> $ActionFileEnableSync off >>>>>> >>>>>> $umask 0007 >>>>>> >>>>>> if $fromhost-ip == "10.10.1.7" and $syslogfacility-text == "local6" >>>>>> and >>>>>> $syslogseverity-text == "debug" and $syslogtag == "TEST1" then >>>>>> action(type="omfile" DirOwner="user1" DirCreateMode="0750" >>>>>> FileCreateMode="0444" File="/rsyslog-data/file1.log"****) >>>>>> >>>>>> if $fromhost-ip == ........ >>>>>> >>>>>> ##############################****# >>>>>> >>>>>> My questions are: >>>>>> 1)how to find the right combination of >>>>>> Main queue ---> Action Queue---> receiver queue on rsyslog server ? >>>>>> 2)Is it possible to increase incoming messages and handle quickly? >>>>>> 3)Why the output log on Centralized Rsyslog are still behind the >>>>>> source >>>>>> log on the client? >>>>>> >>>>>> I've activated the pstats module to understand the queue status on >>>>>> clients and server. >>>>>> After 6 hours running the client has this report >>>>>> ......... >>>>>> 2013-06-03T22:23:16.708288+02:****00 app01 rsyslogd-pstats: action 9 >>>>>> queue: >>>>>> size=2000000 enqueued=9327675 full=6831507 maxqsize=20 >>>>>> 00000 >>>>>> 2013-06-03T22:23:16.708297+02:****00 app01 rsyslogd-pstats: main Q: >>>>>> size=2 >>>>>> enqueued=9648448 full=0 maxqsize=20395 >>>>>> 2013-06-03T22:23:46.708367+02:****00 app01 rsyslogd-pstats: imuxsock: >>>>>> submitted=323414 ratelimit.discarded=0 ratelimit.numratelimi >>>>>> ters=29974 >>>>>> 2013-06-03T22:23:46.708382+02:****00 app01 rsyslogd-pstats: action 9 >>>>>> queue: >>>>>> size=2000000 enqueued=9340578 full=6844410 maxqsize=20 >>>>>> 00000 >>>>>> 2013-06-03T22:23:46.708390+02:****00 app01 rsyslogd-pstats: main Q: >>>>>> size=2164 >>>>>> enqueued=9666464 full=0 maxqsize=20395 >>>>>> 2013-06-03T22:24:16.708923+02:****00 app01 rsyslogd-pstats: imuxsock: >>>>>> submitted=328198 ratelimit.discarded=0 ratelimit.numratelimi >>>>>> ters=29986 >>>>>> 2013-06-03T22:24:16.708941+02:****00 app01 rsyslogd-pstats: action 9 >>>>>> queue: >>>>>> size=2000000 enqueued=9355649 full=6859481 maxqsize=20 >>>>>> 00000 >>>>>> 2013-06-03T22:24:16.708949+02:****00 app01 rsyslogd-pstats: main Q: >>>>>> size=1364 >>>>>> enqueued=9686593 full=0 maxqsize=20395 >>>>>> 2013-06-03T22:24:46.709300+02:****00 app01 rsyslogd-pstats: imuxsock: >>>>>> submitted=333070 ratelimit.discarded=0 ratelimit.numratelimi >>>>>> ters=29997 >>>>>> 2013-06-03T22:24:46.709316+02:****00 app01 rsyslogd-pstats: action 9 >>>>>> queue: >>>>>> size=2000000 enqueued=9365276 full=6869108 maxqsize=20 >>>>>> 00000 >>>>>> 2013-06-03T22:24:46.709323+02:****00 app01 rsyslogd-pstats: main Q: >>>>>> size=2123 >>>>>> enqueued=9702047 full=0 maxqsize=20395 >>>>>> 2013-06-03T22:25:16.709807+02:****00 app01 rsyslogd-pstats: imuxsock: >>>>>> submitted=337951 ratelimit.discarded=0 ratelimit.numratelimi >>>>>> ters=30009 >>>>>> 2013-06-03T22:25:16.709823+02:****00 app01 rsyslogd-pstats: action 9 >>>>>> queue: >>>>>> size=2000000 enqueued=9379492 full=6883324 maxqsize=20 >>>>>> 00000 >>>>>> 2013-06-03T22:25:16.709832+02:****00 app01 rsyslogd-pstats: main Q: >>>>>> size=2 >>>>>> enqueued=9719723 full=0 maxqsize=20395 >>>>>> 2013-06-03T22:25:46.709942+02:****00 app01 rsyslogd-pstats: imuxsock: >>>>>> submitted=343014 ratelimit.discarded=0 ratelimit.numratelimi >>>>>> ters=30021 >>>>>> 2013-06-03T22:25:46.709980+02:****00 app01 rsyslogd-pstats: action 9 >>>>>> queue: >>>>>> size=2000000 enqueued=9389640 full=6893472 maxqsize=20 >>>>>> 00000 >>>>>> >>>>>> >>>>>> and the rsyslog server the following stats >>>>>> ....... >>>>>> 2013-06-03T23:05:15.898682+02:****00 fsp01 rsyslogd-pstats: main Q: >>>>>> size=9978 >>>>>> enqueued=5032165 full=156941 discarded.full=0 discarded.nf=0 >>>>>> maxqsize=10000 >>>>>> 2013-06-03T23:06:56.157199+02:****00 fsp01 rsyslogd-pstats: main Q: >>>>>> size=9987 >>>>>> enqueued=5065134 full=157971 discarded.full=0 discarded.nf=0 >>>>>> maxqsize=10000 >>>>>> 2013-06-03T23:08:30.657673+02:****00 fsp01 rsyslogd-pstats: main Q: >>>>>> size=9972 >>>>>> enqueued=5096315 full=158942 discarded.full=0 discarded.nf=0 >>>>>> maxqsize=10000 >>>>>> 2013-06-03T23:10:09.895850+02:****00 fsp01 rsyslogd-pstats: main Q: >>>>>> size=9986 >>>>>> enqueued=5129162 full=159969 discarded.full=0 discarded.nf=0 >>>>>> maxqsize=10000 >>>>>> 2013-06-03T23:11:42.488505+02:****00 fsp01 rsyslogd-pstats: main Q: >>>>>> size=9973 >>>>>> enqueued=5159935 full=160933 discarded.full=0 discarded.nf=0 >>>>>> maxqsize=10000 >>>>>> 2013-06-03T23:13:23.213800+02:****00 fsp01 rsyslogd-pstats: main Q: >>>>>> size=9973 >>>>>> enqueued=5193246 full=161973 discarded.full=0 discarded.nf=0 >>>>>> maxqsize=10000 >>>>>> 2013-06-03T23:14:58.833570+02:****00 fsp01 rsyslogd-pstats: main Q: >>>>>> size=9970 >>>>>> enqueued=5224922 full=162962 discarded.full=0 discarded.nf=0 >>>>>> maxqsize=10000 >>>>>> 2013-06-03T23:16:35.184133+02:****00 fsp01 rsyslogd-pstats: main Q: >>>>>> size=9975 >>>>>> enqueued=5256863 full=163960 discarded.full=0 discarded.nf=0 >>>>>> maxqsize=10000 >>>>>> 2013-06-03T23:18:13.992958+02:****00 fsp01 rsyslogd-pstats: main Q: >>>>>> size=9991 >>>>>> enqueued=5289392 full=164977 discarded.full=0 discarded.nf=0 >>>>>> maxqsize=10000 >>>>>> 2013-06-03T23:19:52.464473+02:****00 fsp01 rsyslogd-pstats: main Q: >>>>>> size=9942 >>>>>> enqueued=5322013 full=165996 discarded.full=0 discarded.nf=0 >>>>>> maxqsize=10000 >>>>>> >>>>>> 4)Why on the client enqueued value never decrease and full= is always >>>>>> different of 0? >>>>>> >>>>>> I'm planning to distribute the 20 client to 2 process of rsyslog >>>>>> server >>>>>> on different port on the same server but I think your help about the >>>>>> right >>>>>> combination of action queue on client and main queue on server. >>>>>> >>>>>> On rsyslog documentation I've found a lot of info about to handle >>>>>> input >>>>>> queue (# thread, max messages and DequeuBatch) but I dont' know how to >>>>>> rsyslog should better work on receiver server. >>>>>> I've seen that the rsyslog 7.2 process on server use only one CPU each >>>>>> time, while the server has 4CPU and 12GB ram on 64bit O.S. >>>>>> 5)Some option during configuration process can change this behavior ? >>>>>> $ ./configure --prefix=/usr/local/rsyslog7 --enable-gnutls >>>>>> --enable-imfile --enable-omruleset --enable-imptcp >>>>>> --enable-mmnormalize >>>>>> --enable-usertools --enable-imdiag --enable-diagtools >>>>>> --enable-impstats >>>>>> >>>>>> >>>>>> Regards, >>>>>> Roberto. >>>>>> ______________________________****_________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>>>> > >>>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/> >>>>>> > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> ______________________________****_________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>>>> > >>>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/> >>>>>> > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> >>>>>> >>>>> >>>>> ______________________________****_________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>> > >>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/> >>> > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> > ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
