On Thu, 2 Apr 2015, Luis Londoño wrote:
Hi All,
I'm trying to parse CSV logs with rulebase. Here is a sample rule:
prefix=%date:date-rfc3164% %hostname:word% %tag:char-to:,%,
rule=:%receive_time:char-to:,%,%serial_number:char-to:,%,%type:char-to:,%,%subtype:char-to:,%,%fu2:char-to:,%,%generated_time:char-to:,%,%source_ip:char-to:,%,%destination_ip:char-to:,%,%nat_source_ip:char-to:,%,%nat_destination_ip:char-to:,%,%rule_name:c
har-to:,%,%source_user:char-to:,%,%destination_user:char-to:,%,%application:char-to:,%,%virtual_system:char-to:,%,
I'm testing with lognormalizer and its working:
echo "Apr 1 21:09:06 192.168.5.81 1,2015/04/01
21:09:06,001801004232,TRAFFIC,end,1,2015/04/01
21:09:06,192.168.68.33,192.168.0.13,192.0.2.1,192.168.0.13,test,user1,user2,incomplete,test,"
|/usr/lib/lognorm/lognormalizer -r /etc/rsyslog.d/test-traffic.rule -T
-p
[cee@115 virtual_system="test" application="incomplete"
destination_user="user2" source_user="user1" rule_name="test"
nat_destination_ip="192.168.0.13" nat_source_ip="192.0.2.1"
destination_ip="192.168.0.13" source_ip="192.168.68.33"
generated_time="2015/04/01 21:09:06" fu2="1" subtype="end"
type="TRAFFIC" serial_number="001801004232" receive_time="2015/04/01
21:09:06" tag="1" hostname="192.168.5.81" date="Apr 1 21:09:06"]
The problem is that when some field in CSV is empty (two or more
commans joined), the parser fails:
echo "Apr 1 21:09:06 192.168.5.81 1,2015/04/01
21:09:06,001801004232,TRAFFIC,end,1,2015/04/01
21:09:06,192.168.68.33,192.168.0.13,192.0.2.1,192.168.0.13,test,user1,user2,incomplete*,,*"
|/usr/lib/lognorm/lognormalizer -r /etc/rsyslog.d/test-traffic.rule -T
-p
1 unparsable entries
this is one of the failings in the current parser, it doesn't handle this case
well.
It is may be almost possible to use the descent option (in the very latest
release) to
do this.
create a separate file
csv.rb
rule=:,%tail:rest%
rule=:%i:char-to:,%,%tail:rest%
then everywhere in your config that you currently have "char-to:," replace it
with "descent:csv.rb" (path may end up being required)
This won't work as the very last element of the line (you would have to have two
versions of the rule, one ending in , and one ending in the last element)
It will also make your variable names a little odd. instead of $!serial_number
you will have $!serial_number!i
but I think it will work for you.
We are doing some very serious rethinking of this right now, so it is likely to
improve drastically in the next few months.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
