On Tue, 29 Dec 2015, Muhammad Asif wrote:
Date: Tue, 29 Dec 2015 15:06:53 +0500
From: Muhammad Asif <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Can I use multiple stop in filters
Thanks for helping. Now I am using queues at ruleset level. Now my main
queue is empty and ruleset queues never discard in previous two hours.
Before this my action queues start discarding after one minute. What reason
could be?
what do you mean by never discarding?
the queues only loose messages if they are full. When they are able to deliver
messages to the destination, they deliver the message and remove it from the
queue immediatly. Having large queue sizes is an indication that things are
broken.
Secondly now red bold values are often differs. Before it was always
equal. Why is that? Thanks
since I can't see color in my mail client, I'm not sure exactly what you are
talking about, but if you mean the difference between submitted and enqueued for
TCP, that is probably because the option to reset the counters each time is
imprecise and the different counters get reset at slightly different times (this
is a perfomance trade-off, getting exact numbers all the time would
significantly slow rsyslogs processing of normal messages) without resetting the
counters, you may still get a little variation due to sampling the counters at
slightly different times, but over time the numbers will be _very_ close to each
other.
David Lang
Tue Dec 29 14:54:55 2015: imuxsock: origin=imuxsock submitted=0
ratelimit.discarded=0 ratelimit.numratelimiters=0
Tue Dec 29 14:54:55 2015: Network-Flows: origin=core.action
processed=13500 failed=0
suspended=0 suspended.duration=0 resumed=0
Tue Dec 29 14:54:55 2015: Events-on-TCP: origin=core.action processed=*12000
*failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Dec 29 14:54:55 2015: Events-on-UDP: origin=core.action processed=14
failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Dec 29 14:54:55 2015: imudp(*:514): origin=imudp submitted=14
Tue Dec 29 14:54:55 2015: imudp(*:514): origin=imudp submitted=0
Tue Dec 29 14:54:55 2015: imtcp(514): origin=imtcp submitted=*14070*
Tue Dec 29 14:54:55 2015: resource-usage: origin=impstats utime=68071319
stime=24780534 maxrss=1517600 minflt=695465 majflt=6416 inblock=2668032
oublock=3872 nvcsw=613465 nivcsw=10101
Tue Dec 29 14:54:55 2015: Network-Flows: origin=core.queue size=970000
enqueued=13500 full=0 discarded.full=0 discarded.nf=0 maxqsize=970000
Tue Dec 29 14:54:55 2015: Events-on-TCP: origin=core.queue size=700000
enqueued=*13999 *full=0 discarded.full=0 discarded.nf=0 maxqsize=700023
Tue Dec 29 14:54:55 2015: Events-on-UDP: origin=core.queue size=0
enqueued=14 full=0 discarded.full=0 discarded.nf=0 maxqsize=131
Tue Dec 29 14:54:55 2015: main Q: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=6
Tue Dec 29 14:54:55 2015: imudp(w0): origin=imudp called.recvmmsg=9
called.recvmsg=0 msgs.received=14
On Tue, Dec 29, 2015 at 1:07 PM, David Lang <[email protected]> wrote:
you have rulesets for imudp, imtcp, and imfile, but not for pstats,
internal rsyslog messages, and stuff written to /dev/log by applications
I think you need to define queues for the rulesets rather than for each
action (just move all the queue.* things to the ruleset () section)
by the way, since you have the reset turned on on your impstats line, the
numbers are for the prior 10 seconds
David Lang
On Tue, 29 Dec 2015, Muhammad Asif wrote:
Date: Tue, 29 Dec 2015 12:56:30 +0500
From: Muhammad Asif <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Can I use multiple stop in filters
Here is my config file. http://pastebin.com/k4EWRwL7
I am using ruleset tied to input but my main queue still receiving
messages. How can I trouble shoot this issue. Should I remove main queue
configs. Moreover impstats are being reset after each pool time.
Queues stats: http://pastebin.com/asMECzaS
Thanks
On Tue, Dec 29, 2015 at 12:36 PM, David Lang <[email protected]> wrote:
you need to give the config when you ask questions like this. with
impstats the answer could be either way, by default the counters are not
reset, they are a running total since startup, but there is an option to
reset the counters each time they are reported.
On Mon, 28 Dec 2015, Muhammad Asif wrote:
Date: Mon, 28 Dec 2015 18:45:09 +0500
From: Muhammad Asif <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Can I use multiple stop in filters
Dear Sir,
Please have a look on it http://pastebin.com/X2iNWmSh .
Please throw some light.
Mon Dec 28 18:05:58 2015: imtcp(514): origin=imtcp submitted=14101
Mon Dec 28 18:06:08 2015: imtcp(514): origin=imtcp submitted=34825
Mon Dec 28 18:06:19 2015: imtcp(514): origin=imtcp submitted=26688
1- Are these values accumulated or new in 10 seconds on tcp port.
yes (see above)
Mon Dec 28 18:05:58 2015: flows-queue queue: origin=core.queue
size=1000000
enqueued=18007 full=16 discarded.full=7
Mon Dec 28 18:06:08 2015: flows-queue queue: origin=core.queue
size=1000000
enqueued=14007 full=14 discarded.full=7
Mon Dec 28 18:06:19 2015: flows-queue queue: origin=core.queue
size=1000000
enqueued=10008 full=13 discarded.full=8
2- Are enqueued are new messages come into action queue from main queue.
yes
Does it also mean 18007+14007+10008=42022 messages dropped or how many
messages dropped due to discard.full=7 here?
it means 8 were dropped due ot the queue being full, the queue was full
13
times
since later values can be smaller than earlier ones, this looks like it
is
resetting this counter each time it's being reported.
since size is always being reported at the same, very round, value It
looks like you have the queue full each time you are reporting.
3- Messages read from file are also first go to main queue and then come
to
action queue or just come to action queue and then forward.
if you are not using rulesets, things go to the main queue. If you are
using rulesets and have a ruleset tied to an input and have a queue for
that ruleset, that queue is the 'main' queue for that input, the logs
never
touch the MAIN queue.
Thanks
On Mon, Dec 28, 2015 at 5:41 PM, Rainer Gerhards <
[email protected]>
wrote:
Define the queue settings on the ruleset. That's faster and achieves the
same result for this configuration.
You can drop the stop statements. At end of ruleset processing always
stops.
Hth Rainer
Sent from phone, thus brief.
Am 28.12.2015 12:38 schrieb "Muhammad Asif" <[email protected]>:
Sorry I was wrong. ruleset is available in imfile. I am acheiving my
goal
as shown below. Please give you valuable comments.
main_queue(
queue.dequeueBatchSize="4000"
queue.workerthreads="2"
queue.size="2000000"
)
module(load="imfile" PollingInterval="30" )
input(type="imfile" ruleset="flows"
File="/opt/parser/flows/aggregated_flows.csv"
Tag=""
)
ruleset(name="flows"){
action(type="omfwd" target="127.0.0.1" port="5172" protocol="tcp"
name="flows-queue" template="msgonly" queue.size="1000000"
# queue.filename="forwarding" queue.maxdiskspace="1g"
queue.highwatermark="900000" queue.lowwatermark= "500000"
queue.dequeuebatchsize="2000" queue.dequeueslowdown="1000000"
queue.workerthreads="2" queue.type="LinkedList" )
stop
}
input(type="imtcp" port="514" ruleset="events")
ruleset(name="events"){
action(type="omfwd" target="127.0.0.1" port="5170" protocol="tcp"
name="events-queue" template="msgonly" queue.size="1000000"
# queue.filename="forwarding" queue.maxdiskspace="1g"
queue.highwatermark="900000" queue.lowwatermark= "500000"
queue.dequeuebatchsize="2000" queue.dequeueslowdown="1000000"
queue.workerthreads="2" queue.type="LinkedList" )
stop
}
Please answer some queries.
1- Flows taking from csv file also first go to main queue and then
come
to
respective action queue?
2- Is there any better way?
Thanks
On Mon, Dec 28, 2015 at 2:09 PM, Muhammad Asif <[email protected]>
wrote:
Hi David,
As you know ruleset is not available in imfile module then what is
the
best way to deal with logs processing from file and receiving on tcp
port
514 differently and avoid being written in any file even not syslog.
Thanks
On Mon, Dec 28, 2015 at 12:57 PM, David Lang <[email protected]> wrote:
yes, you can use stop as many times as you want.
David Lang
On Mon, 28 Dec 2015, Muhammad Asif wrote:
Date: Mon, 28 Dec 2015 11:19:49 +0500
From: Muhammad Asif <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] Can I use multiple stop in filters
Hi geeks,
Can I use "stop" (To avoid writing in syslog file) in multiple
filters
like
this.
input(type="imptcp" port="514" ruleset="events");
ruleset(name="events"){
action(type="omfwd" target="127.0.0.1" port="5170"
protocol="tcp"
name="events-queue" )
stop
}
module(load="imfile" PollingInterval="30" ruleset="flows")
input(type="imfile" File="/opt/parser/flows/aggregated_flows.csv"
Tag=""
)
ruleset(name="flows"){
action(type="omfwd" target="127.0.0.1" port="5172"
protocol="tcp"
name="flows-queue")
stop
}
Thanks
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
