2016-10-13 16:24 GMT+02:00 Scot Kreienkamp <scot.kreienk...@la-z-boy.com>: > Ideally I'm trying to group on each sending host by the destination of the > data. The main queue will be for local disk write and a secondary queue for > network forwarding so there's no chance it can block the local disk writes. > > As I have it written in my example the best I can come up with is a queue for > each remote port (which is still an improvement over what I have now, so > thanks for that). I am trying to figure out if there's a way to collapse > that further into one network forwarding queue instead of a separate queue > for each remote port, so I end with two queues total. I can't see any way to > do that while still maintaining the multiple destination ports.
I see two ways: 1. (not sure if all modules support this): use a dedicted queue for local logging (ruleset="" param for local input module required) and use the regular main queue for the rest 2. put all forwarding actions into a SINGLE rule set and have some logic inside it to select which action (aka port) you need to call for a given message > > As a secondary concern, I am pushing this config out to a few hundred servers > via puppet. If I go with the example configuration I will be creating 12 > queues times 250 rsyslog senders, so I'm concerned about the possibility of > eating up resources for queues that may never be used on a host. Does > rsyslog consume any resources for a queue that's never called on a linux host? Not noticable, except when using "fixedArray" for the in memory queue. But even then the overhead is very low. HTH Rainer > > > > Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate > One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: 734-384-6403 | | > Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com > -----Original Message----- > From: rsyslog-boun...@lists.adiscon.com > [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, October 13, 2016 9:40 AM > To: rsyslog-users > Subject: Re: [rsyslog] Problem with queues configuration > > 2016-10-13 15:35 GMT+02:00 Scot Kreienkamp <scot.kreienk...@la-z-boy.com>: >> Hi Rainer, >> >> Thanks for your response. I'm still trying to figure out if there's a way >> to simplify my configuration a little more, or if I should leave it as is... >> >> Right now I have different applications forwarding to one of about a dozen >> separate incoming ports on my central rsyslog server. Right now they each >> have individual queues because I didn't want them blocking the main queue, >> which has happened in my environment. If I were to do this, I would have to >> create a ruleset for each port as the actions each ruleset contains would >> have a different destination port. So on each host I would still have about >> a dozen queues. Down from 50 or so on some hosts so it's still better, but >> is there any way to collapse it further into one queue? Does rsyslog use >> any additional resources for a queue that's not called on a linux host? If >> not, I might be better off to leave each port as its own queue like this. > > > I don't really understand what you would prefer. Where would you like > to have dedicated queues for, and which objects should use shared > queues? > > Rainer >> >> Example of my rulesets that I created from your earlier email. >> >> ruleset(name="fwd-monvsyslog-514" >> queue.type="LinkedList" >> queue.size="50000" >> queue.maxDiskSpace="5g" >> queue.discardmark="45000" >> queue.dequeueBatchSize="1024" >> queue.filename="fwd-monvsyslog-514" >> queue.spoolDirectory="/var/spool/rsyslog" >> queue.saveOnShutdown="on" >> >> ){ >> action( >> name="fwd-monvsyslog-514" >> type="omfwd" >> Target="monvsyslog.na.lzb.hq" >> Port="514" >> Protocol="tcp" >> ) >> } >> >> ruleset(name="fwd-monvsyslog-515" >> queue.type="LinkedList" >> queue.size="50000" >> queue.maxDiskSpace="5g" >> queue.discardmark="45000" >> queue.dequeueBatchSize="1024" >> queue.filename="fwd-monvsyslog-515" >> queue.spoolDirectory="/var/spool/rsyslog" >> queue.saveOnShutdown="on" >> >> ){ >> action( >> name="fwd-monvsyslog-514" >> type="omfwd" >> Target="monvsyslog.na.lzb.hq" >> Port="515" >> Protocol="tcp" >> ) >> } >> >> >> >> Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate >> One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: 734-384-6403 | | >> Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com >> -----Original Message----- >> From: rsyslog-boun...@lists.adiscon.com >> [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Thursday, October 13, 2016 8:52 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Problem with queues configuration >> >> sorry, hit wrong key. Here comes the comnplete message ;) >> >> 2016-10-13 14:49 GMT+02:00 Rainer Gerhards <rgerha...@hq.adiscon.com>: >>> 2016-10-13 13:45 GMT+02:00 Angel L. Mateo <ama...@um.es>: >>>> El 11/10/16 a las 15:17, Angel L. Mateo escribió: >>>>> >>>>> El 11/10/16 a las 15:01, Rainer Gerhards escribió: >>>>>> >>>>>> Can you check if the problem persists with the currently supported 8.22.0 >>>>>> version? V7 is very, very old. >>>>>> >>>>> I have to make deeper tests, but it seems to work with 8.22. >>>>> >>>> I can confirm that is working fine with 8.22.0. >>>> >>>> One more question... I have a lot of this rule sending to a remote >>>> syslog depending on different conditions. Is there any way to share the >>>> queue between all these rules? >>> >> The way to do this is to use a ruleset. Instead of doing >> >> action(name="action1" type="omfwd" queue. ...) >> action(name="action2" type="omfwd" queue. ...) >> action(name="action3" type="omfwd" queue. ...) >> >> do >> >> ruleset(name="fwdtodest" queue. ...) { >> action(name="action" type="omfwd" ...) # NO queue params >> } >> >> and replace the action calls with >> >> call fwdtodest >> >> This will make all actions use the single ruleset queue. Note that you >> now have a single connection to the destination and thus possibly >> different timing. But that's probably what you also wanted. >> >> HTH >> Rainer >>>> >>>> >>>> -- >>>> Angel L. Mateo Martínez >>>> Sección de Telemática >>>> Área de Tecnologías de la Información >>>> y las Comunicaciones Aplicadas (ATICA) >>>> http://www.um.es/atica >>>> Tfo: 868889150 >>>> Fax: 868888337 >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>>> LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> >> This message is intended only for the individual or entity to which it is >> addressed. It may contain privileged, confidential information which is >> exempt from disclosure under applicable laws. If you are not the intended >> recipient, you are strictly prohibited from disseminating or distributing >> this information (other than to the intended recipient) or copying this >> information. If you have received this communication in error, please >> notify us immediately by e-mail or by telephone at the above number. Thank >> you. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
