I am not sure if I manage to look at it before vacation, but could you send me a log sample that generates the garbagge? Could you run it under valgrind control? I guess there is some mis-access taking place...
Rainer On Wed, Jul 17, 2013 at 4:03 AM, Todd Mortensen <[email protected]>wrote: > I have made some progress in getting it to work, I removed the leading > space in the rule and it now matches. I totally missed that on log output. > > rule=:%kerntime:word% Denied: IN=%in:word% OUT=%out:word% MAC=%mac:word% > SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number% TOS=%tos:word% > PREC=%prec:word% TTL=%ttl:number% ID=%id:number% %DF:word% > PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number% > WINDOW=%window:number% RES=%res:word% %pkt-type:word% URGP=%urgp:number% > > I did find how to use the iptables type but my results where not very good. > > using this rule my output was has a lot of garbage output in the log, looks > like it wrote out binary data: > > rule=:%kerntime:word% Denied: %iptables:iptables% > > { "IN": "eth0", "OUT": "", "MAC": > "00:30:48:90:cc:a6:00:30:48:da:48:e8:08:00", "SRC": "10.10.10.10", "DST": > "10.10.10.11", "LEN": "60", "TOS": "0x10", "PREC": "0x00", "TTL": "64", > "ID": "10181", "DF": "[*PRESENT*]", "PROTO": "TCP", "SPT": "51765", "DPT": > "444", "WINDOW": "14600", "RES": "0x00", "SYN": "[*PRESENT*]", "URGP": "0", > > "U)@00:30:48:90:cc:a6:00:30:48:da:48:e8:08:00ü«^?xü«^?ÿÿÿÿÿÿÿÿ%\u0003\u0010LENü«^?<80>¸ü«^?%\u0002\u001060ü«^?¼ü«^?%LEN«^?xü«^?@µü«^?%60ü«^?xü«^?¹ü«^?%\býý\u0001p¾ü«^?<80>ºü«^?%\u0003\bLENü«^?<80>»ü«^?%`´ü«^?ð¿ü«^?ðbü«^?%\u0003\u0010TOSü«^?ÿÿÿÿÿÿÿÿ%\u0004\u00100x10«^?ÿÿÿÿÿÿÿÿ%TOS«^?xü«^?¸ü«^?%0x10^?xü«^?ÿÿÿÿÿÿÿÿ%\býý\u0001P¿ü«^?ÿÿÿÿÿÿÿÿ%\u0003\bTOSü«^?»ü«^?%": > "[*PRESENT*]", "kerntime": "[1151929.993763]" } > > Using the full rule I get results like this, nice and clean. > > { "urgp": "0", "pkt-type": "SYN", "res": "0x00", "window": "14600", > "dst-port": "444", "src-port": "51954", "proto": "TCP", "DF": "DF", "id": > "31890", "ttl": "64", "prec": "0x00", "tos": "0x10", "len": "60", "dst-ip": > "10.10.10.10", "src-ip": "10.10.10.11", "mac": > "00:30:48:90:cc:a6:00:30:48:da:48:e8:08:00", "in": "eth0", "kerntime": > "[1152127.460873]" } > > I will continue my journey using the other method for now. > > > On Tue, Jul 16, 2013 at 3:19 PM, Todd Mortensen <[email protected] > >wrote: > > > I am not sure, I honestly stole the examples from the sagan project. > > > > I just tried using the following to test. > > > > rule=: %kerntime:word% Denied: %iptables% > > rule=: [%kerntime:word%] Denied: %iptables% > > > > I also tried adding [] to the above full line as well. Still doesn't > > return any parsed values. > > > > I forgot to mention this is version 7.4.2 on ubuntu 12.04 from my ppa > > > > ii rsyslog-mmnormalize 7.4.2-0ubuntu1ppa2 > > > > > > > > > > On Tue, Jul 16, 2013 at 3:12 PM, David Lang <[email protected]> wrote: > > > >> it looks to me like your ruleset is not matching the rule > >> > >> are you sure that %kerntime:word% includes the square brackets and it > >> shouldn't be [%kerntime:word%] instead? > >> > >> there is a special type 'iptables' that was created explicitly for > >> matching the name=value format of iptables rules. > >> > >> I think your rule can be simplified to something like: > >> > >> rule=: [%kerntime:word%] Denied: %iptables% > >> > >> David Lang > >> > >> On Tue, 16 Jul 2013, Todd Mortensen wrote: > >> > >> Date: Tue, 16 Jul 2013 14:53:36 -0700 > >>> From: Todd Mortensen <[email protected]> > >>> Reply-To: rsyslog-users <[email protected]> > >>> To: rsyslog-users <[email protected]> > >>> Subject: [rsyslog] mmnormalize and iptable logs > >>> > >>> > >>> I am trying my first attempts at using mmnormalize. > >>> > >>> My goal is to parse out my iptable firewall logs into a cee format so I > >>> can > >>> then send them to elasticsearch. > >>> > >>> But so far my logs just show unparsed items. Can anyone shed any light > >>> to > >>> where I am going wrong here. > >>> > >>> Relevant sections from rsyslog.conf > >>> > >>> module(load="mmnormalize") > >>> action(type="mmnormalize" UseRawMsg="off" > >>> ruleBase="/etc/rsyslog.d/**normalize.rb") > >>> > >>> module(load="mmjsonparse") > >>> > >>> > >>> #template(name="cee" type="string" string="%$!all-json%\n") > >>> template(name="cee" type="string" string="%$!%\n") > >>> > >>> *.* { action (name="cee" type="omfile" > >>> Template="cee" file="/var/log/cee.log") } > >>> > >>> I have also tried a couple variations of this but they still don't > match > >>> > >>> normalize.rb > >>> rule=: %kerntime:word% Denied: IN=%in:word% OUT= MAC=%mac:word% > >>> SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number% TOS=%tos:word% > >>> PREC=%prec:word% TTL=%ttl:number% ID=%id:number% %DF:word% > >>> PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number% > >>> WINDOW=%window:number% RES=%res:word% %pkt-type:word% > URGP=%urgp:number% > >>> > >>> I also tried to use the iptales type, but I am not able to find many > docs > >>> on its usage. > >>> > >>> rule=: %kerntime:word% Denied: %iptables:iptables% > >>> > >>> > >>> Sample log line: > >>> > >>> [1084540.211910] Denied: IN=eth0 OUT= > >>> MAC=00:30:48:90:cc:a6:00:30:**48:da:48:e8:08:00 SRC=10.10.10.10 > >>> DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF PROTO=TCP > >>> SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0 > >>> > >>> > >>> This is what I see in my cee logfile and it shows unpared-data. > >>> > >>> # { "originalmsg": "[1084540.211910] Denied: IN=eth0 OUT= > >>> MAC=00:30:48:90:cc:a6:00:30:**48:da:48:e8:08:00 SRC=10.10.10.10 > >>> DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF PROTO=TCP > >>> SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0 ", "unparsed-data": > >>> "[1084540.211910] Denied: IN=eth0 OUT= > >>> MAC=00:30:48:90:cc:a6:00:30:**48:da:48:e8:08:00 SRC=10.10.10.10 > >>> DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF PROTO=TCP > >>> SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0 " } > >>> ______________________________**_________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>> http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >>> ______________________________**_________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >> http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
