Here is my config file. http://pastebin.com/k4EWRwL7
I am using ruleset tied to input but my main queue still receiving messages. How can I trouble shoot this issue. Should I remove main queue configs. Moreover impstats are being reset after each pool time. Queues stats: http://pastebin.com/asMECzaS Thanks On Tue, Dec 29, 2015 at 12:36 PM, David Lang <[email protected]> wrote: > you need to give the config when you ask questions like this. with > impstats the answer could be either way, by default the counters are not > reset, they are a running total since startup, but there is an option to > reset the counters each time they are reported. > > On Mon, 28 Dec 2015, Muhammad Asif wrote: > > Date: Mon, 28 Dec 2015 18:45:09 +0500 >> From: Muhammad Asif <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] Can I use multiple stop in filters >> >> Dear Sir, >> >> Please have a look on it http://pastebin.com/X2iNWmSh . >> Please throw some light. >> Mon Dec 28 18:05:58 2015: imtcp(514): origin=imtcp submitted=14101 >> Mon Dec 28 18:06:08 2015: imtcp(514): origin=imtcp submitted=34825 >> Mon Dec 28 18:06:19 2015: imtcp(514): origin=imtcp submitted=26688 >> >> 1- Are these values accumulated or new in 10 seconds on tcp port. >> > > > yes (see above) > > >> Mon Dec 28 18:05:58 2015: flows-queue queue: origin=core.queue >> size=1000000 >> enqueued=18007 full=16 discarded.full=7 >> Mon Dec 28 18:06:08 2015: flows-queue queue: origin=core.queue >> size=1000000 >> enqueued=14007 full=14 discarded.full=7 >> Mon Dec 28 18:06:19 2015: flows-queue queue: origin=core.queue >> size=1000000 >> enqueued=10008 full=13 discarded.full=8 >> >> 2- Are enqueued are new messages come into action queue from main queue. >> > > yes > > Does it also mean 18007+14007+10008=42022 messages dropped or how many >> messages dropped due to discard.full=7 here? >> > > it means 8 were dropped due ot the queue being full, the queue was full 13 > times > > since later values can be smaller than earlier ones, this looks like it is > resetting this counter each time it's being reported. > > since size is always being reported at the same, very round, value It > looks like you have the queue full each time you are reporting. > > 3- Messages read from file are also first go to main queue and then come to >> action queue or just come to action queue and then forward. >> > > if you are not using rulesets, things go to the main queue. If you are > using rulesets and have a ruleset tied to an input and have a queue for > that ruleset, that queue is the 'main' queue for that input, the logs never > touch the MAIN queue. > > > Thanks >> >> >> >> On Mon, Dec 28, 2015 at 5:41 PM, Rainer Gerhards < >> [email protected]> >> wrote: >> >> Define the queue settings on the ruleset. That's faster and achieves the >>> same result for this configuration. >>> >>> You can drop the stop statements. At end of ruleset processing always >>> stops. >>> >>> Hth Rainer >>> >>> Sent from phone, thus brief. >>> Am 28.12.2015 12:38 schrieb "Muhammad Asif" <[email protected]>: >>> >>> Sorry I was wrong. ruleset is available in imfile. I am acheiving my goal >>>> as shown below. Please give you valuable comments. >>>> >>>> >>>> main_queue( >>>> queue.dequeueBatchSize="4000" >>>> queue.workerthreads="2" >>>> queue.size="2000000" >>>> ) >>>> >>>> >>>> module(load="imfile" PollingInterval="30" ) >>>> input(type="imfile" ruleset="flows" >>>> File="/opt/parser/flows/aggregated_flows.csv" >>>> Tag="" >>>> ) >>>> >>>> >>>> ruleset(name="flows"){ >>>> action(type="omfwd" target="127.0.0.1" port="5172" protocol="tcp" >>>> name="flows-queue" template="msgonly" queue.size="1000000" >>>> # queue.filename="forwarding" queue.maxdiskspace="1g" >>>> queue.highwatermark="900000" queue.lowwatermark= "500000" >>>> queue.dequeuebatchsize="2000" queue.dequeueslowdown="1000000" >>>> queue.workerthreads="2" queue.type="LinkedList" ) >>>> stop >>>> } >>>> >>>> >>>> input(type="imtcp" port="514" ruleset="events") >>>> >>>> ruleset(name="events"){ >>>> >>>> action(type="omfwd" target="127.0.0.1" port="5170" protocol="tcp" >>>> name="events-queue" template="msgonly" queue.size="1000000" >>>> # queue.filename="forwarding" queue.maxdiskspace="1g" >>>> queue.highwatermark="900000" queue.lowwatermark= "500000" >>>> queue.dequeuebatchsize="2000" queue.dequeueslowdown="1000000" >>>> queue.workerthreads="2" queue.type="LinkedList" ) >>>> >>>> stop >>>> } >>>> >>>> Please answer some queries. >>>> 1- Flows taking from csv file also first go to main queue and then come >>>> >>> to >>> >>>> respective action queue? >>>> 2- Is there any better way? >>>> >>>> Thanks >>>> >>>> >>>> On Mon, Dec 28, 2015 at 2:09 PM, Muhammad Asif <[email protected]> >>>> wrote: >>>> >>>> Hi David, >>>>> >>>>> As you know ruleset is not available in imfile module then what is the >>>>> best way to deal with logs processing from file and receiving on tcp >>>>> >>>> port >>> >>>> 514 differently and avoid being written in any file even not syslog. >>>>> >>>>> Thanks >>>>> >>>>> On Mon, Dec 28, 2015 at 12:57 PM, David Lang <[email protected]> wrote: >>>>> >>>>> yes, you can use stop as many times as you want. >>>>>> >>>>>> David Lang >>>>>> >>>>>> On Mon, 28 Dec 2015, Muhammad Asif wrote: >>>>>> >>>>>> Date: Mon, 28 Dec 2015 11:19:49 +0500 >>>>>> >>>>>>> From: Muhammad Asif <[email protected]> >>>>>>> Reply-To: rsyslog-users <[email protected]> >>>>>>> To: rsyslog-users <[email protected]> >>>>>>> Subject: [rsyslog] Can I use multiple stop in filters >>>>>>> >>>>>>> >>>>>>> Hi geeks, >>>>>>> >>>>>>> Can I use "stop" (To avoid writing in syslog file) in multiple >>>>>>> >>>>>> filters >>> >>>> like >>>>>>> this. >>>>>>> >>>>>>> input(type="imptcp" port="514" ruleset="events"); >>>>>>> >>>>>>> >>>>>>> >>>>>>> ruleset(name="events"){ >>>>>>> action(type="omfwd" target="127.0.0.1" port="5170" protocol="tcp" >>>>>>> name="events-queue" ) >>>>>>> >>>>>>> stop >>>>>>> } >>>>>>> >>>>>>> >>>>>>> >>>>>>> module(load="imfile" PollingInterval="30" ruleset="flows") >>>>>>> >>>>>>> input(type="imfile" File="/opt/parser/flows/aggregated_flows.csv" >>>>>>> >>>>>>> Tag="" >>>>>>> >>>>>>> ) >>>>>>> >>>>>>> ruleset(name="flows"){ >>>>>>> action(type="omfwd" target="127.0.0.1" port="5172" protocol="tcp" >>>>>>> name="flows-queue") >>>>>>> >>>>>>> stop >>>>>>> } >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> >>>>>> myriad >>>> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>> >>>>>> you >>> >>>> DON'T LIKE THAT. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> >>>>> myriad >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
