Ideally I'm trying to group on each sending host by the destination of the 
data.  The main queue will be for local disk write and a secondary queue for 
network forwarding so there's no chance it can block the local disk writes.

As I have it written in my example the best I can come up with is a queue for 
each remote port (which is still an improvement over what I have now, so thanks 
for that).  I am trying to figure out if there's a way to collapse that further 
into one network forwarding queue instead of a separate queue for each remote 
port, so I end with two queues total.  I can't see any way to do that while 
still maintaining the multiple destination ports.

As a secondary concern, I am pushing this config out to a few hundred servers 
via puppet.  If I go with the example configuration I will be creating 12 
queues times 250 rsyslog senders, so I'm concerned about the possibility of 
eating up resources for queues that may never be used on a host.  Does rsyslog 
consume any resources for a queue that's never called on a linux host?



Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
-----Original Message-----
From: rsyslog-boun...@lists.adiscon.com 
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer Gerhards
Sent: Thursday, October 13, 2016 9:40 AM
To: rsyslog-users
Subject: Re: [rsyslog] Problem with queues configuration

2016-10-13 15:35 GMT+02:00 Scot Kreienkamp <scot.kreienk...@la-z-boy.com>:
> Hi Rainer,
>
> Thanks for your response.  I'm still trying to figure out if there's a way to 
> simplify my configuration a little more, or if I should leave it as is...
>
> Right now I have different applications forwarding to one of about a dozen 
> separate incoming ports on my central rsyslog server.  Right now they each 
> have individual queues because I didn't want them blocking the main queue, 
> which has happened in my environment.  If I were to do this, I would have to 
> create a ruleset for each port as the actions each ruleset contains would 
> have a different destination port.  So on each host I would still have about 
> a dozen queues.  Down from 50 or so on some hosts so it's still better, but 
> is there any way to collapse it further into one queue?  Does rsyslog use any 
> additional resources for a queue that's not called on a linux host?  If not, 
> I might be better off to leave each port as its own queue like this.


I don't really understand what you would prefer. Where would you like
to have dedicated queues for, and which objects should use shared
queues?

Rainer
>
> Example of my rulesets that I created from your earlier email.
>
> ruleset(name="fwd-monvsyslog-514"
>                 queue.type="LinkedList"
>                 queue.size="50000"
>                 queue.maxDiskSpace="5g"
>                 queue.discardmark="45000"
>                 queue.dequeueBatchSize="1024"
>                 queue.filename="fwd-monvsyslog-514"
>                 queue.spoolDirectory="/var/spool/rsyslog"
>                 queue.saveOnShutdown="on"
>
>         ){
>         action(
>                 name="fwd-monvsyslog-514"
>                 type="omfwd"
>                 Target="monvsyslog.na.lzb.hq"
>                 Port="514"
>                 Protocol="tcp"
>         )
> }
>
> ruleset(name="fwd-monvsyslog-515"
>                 queue.type="LinkedList"
>                 queue.size="50000"
>                 queue.maxDiskSpace="5g"
>                 queue.discardmark="45000"
>                 queue.dequeueBatchSize="1024"
>                 queue.filename="fwd-monvsyslog-515"
>                 queue.spoolDirectory="/var/spool/rsyslog"
>                 queue.saveOnShutdown="on"
>
>         ){
>         action(
>                 name="fwd-monvsyslog-514"
>                 type="omfwd"
>                 Target="monvsyslog.na.lzb.hq"
>                 Port="515"
>                 Protocol="tcp"
>         )
> }
>
>
>
> Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
> One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
> Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
> -----Original Message-----
> From: rsyslog-boun...@lists.adiscon.com 
> [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Thursday, October 13, 2016 8:52 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problem with queues configuration
>
> sorry, hit wrong key. Here comes the comnplete message ;)
>
> 2016-10-13 14:49 GMT+02:00 Rainer Gerhards <rgerha...@hq.adiscon.com>:
>> 2016-10-13 13:45 GMT+02:00 Angel L. Mateo <ama...@um.es>:
>>> El 11/10/16 a las 15:17, Angel L. Mateo escribió:
>>>>
>>>> El 11/10/16 a las 15:01, Rainer Gerhards escribió:
>>>>>
>>>>> Can you check if the problem persists with the currently supported 8.22.0
>>>>> version? V7 is very, very old.
>>>>>
>>>>     I have to make deeper tests, but it seems to work with 8.22.
>>>>
>>>         I can confirm that is working fine with 8.22.0.
>>>
>>>         One more question... I have a lot of this rule sending to a remote
>>> syslog depending on different conditions. Is there any way to share the
>>> queue between all these rules?
>>
> The way to do this is to use a ruleset. Instead of doing
>
> action(name="action1" type="omfwd" queue. ...)
> action(name="action2" type="omfwd" queue. ...)
> action(name="action3" type="omfwd" queue. ...)
>
> do
>
> ruleset(name="fwdtodest" queue. ...) {
>    action(name="action" type="omfwd" ...) # NO queue params
> }
>
> and replace the action calls with
>
> call fwdtodest
>
> This will make all actions use the single ruleset queue. Note that you
> now have a single connection to the destination and thus possibly
> different timing. But that's probably what you also wanted.
>
> HTH
> Rainer
>>>
>>>
>>> --
>>> Angel L. Mateo Martínez
>>> Sección de Telemática
>>> Área de Tecnologías de la Información
>>> y las Comunicaciones Aplicadas (ATICA)
>>> http://www.um.es/atica
>>> Tfo: 868889150
>>> Fax: 868888337
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
>
> This message is intended only for the individual or entity to which it is 
> addressed.  It may contain privileged, confidential information which is 
> exempt from disclosure under applicable laws.  If you are not the intended 
> recipient, you are strictly prohibited from disseminating or distributing 
> this information (other than to the intended recipient) or copying this 
> information.  If you have received this communication in error, please notify 
> us immediately by e-mail or by telephone at the above number. Thank you.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to