I have one centralized collector, and it was having trouble keeping up. It normally runs 20-30 meg (combined in and out) during the day, bursting as high as 90 meg. It has 228 if statements, mostly checking syslogtag, but some checking sending hostname, times, etc. To eliminate some processing time I changed the higher traffic apps to come in on their own dedicated ports so I could bind a much smaller ruleset to the inputs, and everything else comes in on the default port. So now the largest number of if statements in a ruleset is 49 with many being 15 or less.
My original intent behind the question was not because I was using a large amount of ports, but so I could group everything outbound from the clients into one outbound queue and have something reusable by selecting a new port instead of writing a new ruleset, actions, and queues for every outbound port. Welcome back :-) Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: 734-384-6403 | | Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com -----Original Message----- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang Sent: Thursday, October 27, 2016 2:49 PM To: rsyslog-users Subject: Re: [rsyslog] Problem with queues configuration no, you cannot use a variable for a port. catching up on this, I'm trying to understand why you are using so many different ports. Each message has the original hostname in it, and you can split the logs on that at the central system rather easily, why split it on the sending side to different ports? If the receiver can't keep up, having lots of different queues on the sender is just going to mean you have a lot of queues filling up, the order that they then drain is rather unpredictable. David Lang On Fri, 21 Oct 2016, Scot Kreienkamp wrote: > Would something like this work? I wasn't sure a config file variable could > be used in the action context. > > > ruleset(name="fwd-centralsyslog-customport" > queue.type="LinkedList" > queue.size="50000" > queue.maxDiskSpace="1g" > queue.discardmark="45000" > queue.dequeueBatchSize="1024" > queue.filename="fwd-monvsyslog-514" > queue.spoolDirectory="/var/spool/rsyslog" > queue.saveOnShutdown="on" > ){ > action( > name="fwd-syslog-collector" > type="omfwd" > Target="monvsyslog " > Port="$!remoteport" > Protocol="tcp" > ) > } > > > > if $syslogtag == 'Ecomm:' then { > set $!remoteport = "1531"; > call fwd-centralsyslog-customport > stop > } > > > > > Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate > One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: 734-384-6403 | | > Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com > -----Original Message----- > From: rsyslog-boun...@lists.adiscon.com > [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, October 13, 2016 8:52 AM > To: rsyslog-users > Subject: Re: [rsyslog] Problem with queues configuration > > sorry, hit wrong key. Here comes the comnplete message ;) > > 2016-10-13 14:49 GMT+02:00 Rainer Gerhards <rgerha...@hq.adiscon.com>: >> 2016-10-13 13:45 GMT+02:00 Angel L. Mateo <ama...@um.es>: >>> El 11/10/16 a las 15:17, Angel L. Mateo escribió: >>>> >>>> El 11/10/16 a las 15:01, Rainer Gerhards escribió: >>>>> >>>>> Can you check if the problem persists with the currently supported 8.22.0 >>>>> version? V7 is very, very old. >>>>> >>>> I have to make deeper tests, but it seems to work with 8.22. >>>> >>> I can confirm that is working fine with 8.22.0. >>> >>> One more question... I have a lot of this rule sending to a remote >>> syslog depending on different conditions. Is there any way to share the >>> queue between all these rules? >> > The way to do this is to use a ruleset. Instead of doing > > action(name="action1" type="omfwd" queue. ...) > action(name="action2" type="omfwd" queue. ...) > action(name="action3" type="omfwd" queue. ...) > > do > > ruleset(name="fwdtodest" queue. ...) { > action(name="action" type="omfwd" ...) # NO queue params > } > > and replace the action calls with > > call fwdtodest > > This will make all actions use the single ruleset queue. Note that you > now have a single connection to the destination and thus possibly > different timing. But that's probably what you also wanted. > > HTH > Rainer >>> >>> >>> -- >>> Angel L. Mateo Martínez >>> Sección de Telemática >>> Área de Tecnologías de la Información >>> y las Comunicaciones Aplicadas (ATICA) >>> http://www.um.es/atica >>> Tfo: 868889150 >>> Fax: 868888337 >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > > This message is intended only for the individual or entity to which it is > addressed. It may contain privileged, confidential information which is > exempt from disclosure under applicable laws. If you are not the intended > recipient, you are strictly prohibited from disseminating or distributing > this information (other than to the intended recipient) or copying this > information. If you have received this communication in error, please notify > us immediately by e-mail or by telephone at the above number. Thank you. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
