Thanks for helping. Now I am using queues at ruleset level. Now my main queue is empty and ruleset queues never discard in previous two hours. Before this my action queues start discarding after one minute. What reason could be? Secondly now red bold values are often differs. Before it was always equal. Why is that? Thanks
Tue Dec 29 14:54:55 2015: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0 Tue Dec 29 14:54:55 2015: Network-Flows: origin=core.action processed=13500 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Dec 29 14:54:55 2015: Events-on-TCP: origin=core.action processed=*12000 *failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Dec 29 14:54:55 2015: Events-on-UDP: origin=core.action processed=14 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Dec 29 14:54:55 2015: imudp(*:514): origin=imudp submitted=14 Tue Dec 29 14:54:55 2015: imudp(*:514): origin=imudp submitted=0 Tue Dec 29 14:54:55 2015: imtcp(514): origin=imtcp submitted=*14070* Tue Dec 29 14:54:55 2015: resource-usage: origin=impstats utime=68071319 stime=24780534 maxrss=1517600 minflt=695465 majflt=6416 inblock=2668032 oublock=3872 nvcsw=613465 nivcsw=10101 Tue Dec 29 14:54:55 2015: Network-Flows: origin=core.queue size=970000 enqueued=13500 full=0 discarded.full=0 discarded.nf=0 maxqsize=970000 Tue Dec 29 14:54:55 2015: Events-on-TCP: origin=core.queue size=700000 enqueued=*13999 *full=0 discarded.full=0 discarded.nf=0 maxqsize=700023 Tue Dec 29 14:54:55 2015: Events-on-UDP: origin=core.queue size=0 enqueued=14 full=0 discarded.full=0 discarded.nf=0 maxqsize=131 Tue Dec 29 14:54:55 2015: main Q: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=6 Tue Dec 29 14:54:55 2015: imudp(w0): origin=imudp called.recvmmsg=9 called.recvmsg=0 msgs.received=14 On Tue, Dec 29, 2015 at 1:07 PM, David Lang <[email protected]> wrote: > you have rulesets for imudp, imtcp, and imfile, but not for pstats, > internal rsyslog messages, and stuff written to /dev/log by applications > > I think you need to define queues for the rulesets rather than for each > action (just move all the queue.* things to the ruleset () section) > > by the way, since you have the reset turned on on your impstats line, the > numbers are for the prior 10 seconds > > David Lang > > > On Tue, 29 Dec 2015, Muhammad Asif wrote: > > Date: Tue, 29 Dec 2015 12:56:30 +0500 >> >> From: Muhammad Asif <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] Can I use multiple stop in filters >> >> Here is my config file. http://pastebin.com/k4EWRwL7 >> >> I am using ruleset tied to input but my main queue still receiving >> messages. How can I trouble shoot this issue. Should I remove main queue >> configs. Moreover impstats are being reset after each pool time. >> >> Queues stats: http://pastebin.com/asMECzaS >> >> Thanks >> >> On Tue, Dec 29, 2015 at 12:36 PM, David Lang <[email protected]> wrote: >> >> you need to give the config when you ask questions like this. with >>> impstats the answer could be either way, by default the counters are not >>> reset, they are a running total since startup, but there is an option to >>> reset the counters each time they are reported. >>> >>> On Mon, 28 Dec 2015, Muhammad Asif wrote: >>> >>> Date: Mon, 28 Dec 2015 18:45:09 +0500 >>> >>>> From: Muhammad Asif <[email protected]> >>>> Reply-To: rsyslog-users <[email protected]> >>>> To: rsyslog-users <[email protected]> >>>> Subject: Re: [rsyslog] Can I use multiple stop in filters >>>> >>>> Dear Sir, >>>> >>>> Please have a look on it http://pastebin.com/X2iNWmSh . >>>> Please throw some light. >>>> Mon Dec 28 18:05:58 2015: imtcp(514): origin=imtcp submitted=14101 >>>> Mon Dec 28 18:06:08 2015: imtcp(514): origin=imtcp submitted=34825 >>>> Mon Dec 28 18:06:19 2015: imtcp(514): origin=imtcp submitted=26688 >>>> >>>> 1- Are these values accumulated or new in 10 seconds on tcp port. >>>> >>>> >>> >>> yes (see above) >>> >>> >>> Mon Dec 28 18:05:58 2015: flows-queue queue: origin=core.queue >>>> size=1000000 >>>> enqueued=18007 full=16 discarded.full=7 >>>> Mon Dec 28 18:06:08 2015: flows-queue queue: origin=core.queue >>>> size=1000000 >>>> enqueued=14007 full=14 discarded.full=7 >>>> Mon Dec 28 18:06:19 2015: flows-queue queue: origin=core.queue >>>> size=1000000 >>>> enqueued=10008 full=13 discarded.full=8 >>>> >>>> 2- Are enqueued are new messages come into action queue from main queue. >>>> >>>> >>> yes >>> >>> Does it also mean 18007+14007+10008=42022 messages dropped or how many >>> >>>> messages dropped due to discard.full=7 here? >>>> >>>> >>> it means 8 were dropped due ot the queue being full, the queue was full >>> 13 >>> times >>> >>> since later values can be smaller than earlier ones, this looks like it >>> is >>> resetting this counter each time it's being reported. >>> >>> since size is always being reported at the same, very round, value It >>> looks like you have the queue full each time you are reporting. >>> >>> 3- Messages read from file are also first go to main queue and then come >>> to >>> >>>> action queue or just come to action queue and then forward. >>>> >>>> >>> if you are not using rulesets, things go to the main queue. If you are >>> using rulesets and have a ruleset tied to an input and have a queue for >>> that ruleset, that queue is the 'main' queue for that input, the logs >>> never >>> touch the MAIN queue. >>> >>> >>> Thanks >>> >>>> >>>> >>>> >>>> On Mon, Dec 28, 2015 at 5:41 PM, Rainer Gerhards < >>>> [email protected]> >>>> wrote: >>>> >>>> Define the queue settings on the ruleset. That's faster and achieves the >>>> >>>>> same result for this configuration. >>>>> >>>>> You can drop the stop statements. At end of ruleset processing always >>>>> stops. >>>>> >>>>> Hth Rainer >>>>> >>>>> Sent from phone, thus brief. >>>>> Am 28.12.2015 12:38 schrieb "Muhammad Asif" <[email protected]>: >>>>> >>>>> Sorry I was wrong. ruleset is available in imfile. I am acheiving my >>>>> goal >>>>> >>>>>> as shown below. Please give you valuable comments. >>>>>> >>>>>> >>>>>> main_queue( >>>>>> queue.dequeueBatchSize="4000" >>>>>> queue.workerthreads="2" >>>>>> queue.size="2000000" >>>>>> ) >>>>>> >>>>>> >>>>>> module(load="imfile" PollingInterval="30" ) >>>>>> input(type="imfile" ruleset="flows" >>>>>> File="/opt/parser/flows/aggregated_flows.csv" >>>>>> Tag="" >>>>>> ) >>>>>> >>>>>> >>>>>> ruleset(name="flows"){ >>>>>> action(type="omfwd" target="127.0.0.1" port="5172" protocol="tcp" >>>>>> name="flows-queue" template="msgonly" queue.size="1000000" >>>>>> # queue.filename="forwarding" queue.maxdiskspace="1g" >>>>>> queue.highwatermark="900000" queue.lowwatermark= "500000" >>>>>> queue.dequeuebatchsize="2000" queue.dequeueslowdown="1000000" >>>>>> queue.workerthreads="2" queue.type="LinkedList" ) >>>>>> stop >>>>>> } >>>>>> >>>>>> >>>>>> input(type="imtcp" port="514" ruleset="events") >>>>>> >>>>>> ruleset(name="events"){ >>>>>> >>>>>> action(type="omfwd" target="127.0.0.1" port="5170" protocol="tcp" >>>>>> name="events-queue" template="msgonly" queue.size="1000000" >>>>>> # queue.filename="forwarding" queue.maxdiskspace="1g" >>>>>> queue.highwatermark="900000" queue.lowwatermark= "500000" >>>>>> queue.dequeuebatchsize="2000" queue.dequeueslowdown="1000000" >>>>>> queue.workerthreads="2" queue.type="LinkedList" ) >>>>>> >>>>>> stop >>>>>> } >>>>>> >>>>>> Please answer some queries. >>>>>> 1- Flows taking from csv file also first go to main queue and then >>>>>> come >>>>>> >>>>>> to >>>>> >>>>> respective action queue? >>>>>> 2- Is there any better way? >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>> On Mon, Dec 28, 2015 at 2:09 PM, Muhammad Asif <[email protected]> >>>>>> wrote: >>>>>> >>>>>> Hi David, >>>>>> >>>>>>> >>>>>>> As you know ruleset is not available in imfile module then what is >>>>>>> the >>>>>>> best way to deal with logs processing from file and receiving on tcp >>>>>>> >>>>>>> port >>>>>> >>>>> >>>>> 514 differently and avoid being written in any file even not syslog. >>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> On Mon, Dec 28, 2015 at 12:57 PM, David Lang <[email protected]> wrote: >>>>>>> >>>>>>> yes, you can use stop as many times as you want. >>>>>>> >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>> On Mon, 28 Dec 2015, Muhammad Asif wrote: >>>>>>>> >>>>>>>> Date: Mon, 28 Dec 2015 11:19:49 +0500 >>>>>>>> >>>>>>>> From: Muhammad Asif <[email protected]> >>>>>>>>> Reply-To: rsyslog-users <[email protected]> >>>>>>>>> To: rsyslog-users <[email protected]> >>>>>>>>> Subject: [rsyslog] Can I use multiple stop in filters >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi geeks, >>>>>>>>> >>>>>>>>> Can I use "stop" (To avoid writing in syslog file) in multiple >>>>>>>>> >>>>>>>>> filters >>>>>>>> >>>>>>> >>>>> like >>>>>> >>>>>>> this. >>>>>>>>> >>>>>>>>> input(type="imptcp" port="514" ruleset="events"); >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ruleset(name="events"){ >>>>>>>>> action(type="omfwd" target="127.0.0.1" port="5170" >>>>>>>>> protocol="tcp" >>>>>>>>> name="events-queue" ) >>>>>>>>> >>>>>>>>> stop >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> module(load="imfile" PollingInterval="30" ruleset="flows") >>>>>>>>> >>>>>>>>> input(type="imfile" File="/opt/parser/flows/aggregated_flows.csv" >>>>>>>>> >>>>>>>>> Tag="" >>>>>>>>> >>>>>>>>> ) >>>>>>>>> >>>>>>>>> ruleset(name="flows"){ >>>>>>>>> action(type="omfwd" target="127.0.0.1" port="5172" >>>>>>>>> protocol="tcp" >>>>>>>>> name="flows-queue") >>>>>>>>> >>>>>>>>> stop >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>> >>>>>>>>> myriad >>>>>>>> >>>>>>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>> >>>>>>>> >>>>>>>>> you >>>>>>>> >>>>>>> >>>>> DON'T LIKE THAT. >>>>>> >>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> >>>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>> >>>>>>>> myriad >>>>>>> >>>>>> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> >>>>>>> DON'T LIKE THAT. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> _______________________________________________ >>>>>> >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>>> _______________________________________________ >>>>> >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>>> >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
