Hello, I'm going to become mad with the rest parser, even in the latest liblognorm release. I just want to normalize this very basic message using the "rest" field-type:
echo "<86>Jun 1 12:37:41 server sshd[23795]: Failed password for root from ::1 port 54849 ssh2" | lognormalizer -r /etc/name.rb This is my ruleset: --------------- prefix=<%prio:number%>%time:date-rfc3164% %host:word% rule=:sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word%%-:rest% rule=:sshd%-:rest% --------------- I use this alternatively: --------------- rule=:<%prio:number%>%time:date-rfc3164% %host:word% sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word% %-:rest% --------------- None of them work. Result is always: [cee@115 originalmsg="<86>Jun 1 12:37:41 server sshd[23795\]: Failed password for root from ::1 port 54849 ssh2" unparsed-data=" ssh2"] I installed the latest release, see here: [root@server ~]# yum info liblognorm1.x86_64 Loaded plugins: fastestmirror, presto Loading mirror speeds from cached hostfile * base: ftp.wrz.de * extras: centos.bio.lmu.de * updates: centos.bio.lmu.de Installed Packages Name : liblognorm1 Arch : x86_64 Version : 1.1.2 Release : 1.el6 Size : 103 k Repo : installed >From repo : rsyslog-v8-stable Summary : Fast samples-based log normalization library URL : http://www.liblognorm.com License : LGPLv2+ Description : Briefly described, liblognorm is a tool to normalize log data. : : People who need to take a look at logs often have a common problem. Logs from : different machines (from different vendors) usually have different formats for : their logs. Even if it is the same type of log (e.g. from firewalls), the log : entries are so different, that it is pretty hard to read these. This is where : liblognorm comes into the game. With this tool you can normalize all your logs. : All you need is liblognorm and its dependencies and a sample database that fits : the logs you want to normalize. Debug-Output: [root@bug ~]# echo "<86>Jun 1 12:37:41 server sshd[23795]: Failed password for root from ::1 port 54849 ssh2" | lognormalizer -r /etc/test3.rb -v liblognorm: read sample line: 'prefix=<%prio:number%>%time:date-rfc3164% %host:word% ' liblognorm: read sample line: 'rule=:sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word%%-:rest%' liblognorm: sample line to add: ':sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word%%-:rest%' liblognorm: addSampToTree 0 of 167 liblognorm: parsed literal: '<' liblognorm: buildPTree: begin at 0x8b0060, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 1, offs 0 liblognorm: addSampToTree 1 of 167 liblognorm: parsed field: 'prio' liblognorm: got new subtree 0x8b0df0 liblognorm: prev subtree 0x8b0060 liblognorm: new subtree 0x8b0df0 liblognorm: addSampToTree 14 of 167 liblognorm: parsed literal: '>' liblognorm: buildPTree: begin at 0x8b0df0, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 1, offs 0 liblognorm: addSampToTree 15 of 167 liblognorm: parsed field: 'time' liblognorm: got new subtree 0x8b16a0 liblognorm: prev subtree 0x8b0df0 liblognorm: new subtree 0x8b16a0 liblognorm: addSampToTree 34 of 167 liblognorm: parsed literal: ' ' liblognorm: buildPTree: begin at 0x8b16a0, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 1, offs 0 liblognorm: addSampToTree 35 of 167 liblognorm: parsed field: 'host' liblognorm: got new subtree 0x8b1f50 liblognorm: prev subtree 0x8b16a0 liblognorm: new subtree 0x8b1f50 liblognorm: addSampToTree 46 of 167 liblognorm: parsed literal: ' sshd[' liblognorm: buildPTree: begin at 0x8b1f50, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 6, offs 0 liblognorm: addSampToTree 52 of 167 liblognorm: parsed field: 'procid' liblognorm: got new subtree 0x8b2800 liblognorm: prev subtree 0x8b1f50 liblognorm: new subtree 0x8b2800 liblognorm: addSampToTree 67 of 167 liblognorm: parsed literal: ']: Failed ' liblognorm: buildPTree: begin at 0x8b2800, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 10, offs 0 liblognorm: addSampToTree 77 of 167 liblognorm: parsed field: 'user_authmethod' liblognorm: got new subtree 0x8b30b0 liblognorm: prev subtree 0x8b2800 liblognorm: new subtree 0x8b30b0 liblognorm: addSampToTree 99 of 167 liblognorm: parsed literal: ' for ' liblognorm: buildPTree: begin at 0x8b30b0, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 5, offs 0 liblognorm: addSampToTree 104 of 167 liblognorm: parsed field: 'username' liblognorm: got new subtree 0x8b3960 liblognorm: prev subtree 0x8b30b0 liblognorm: new subtree 0x8b3960 liblognorm: addSampToTree 119 of 167 liblognorm: parsed literal: ' from ' liblognorm: buildPTree: begin at 0x8b3960, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 6, offs 0 liblognorm: addSampToTree 125 of 167 liblognorm: parsed field: 'src_ip' liblognorm: got new subtree 0x8b4210 liblognorm: prev subtree 0x8b3960 liblognorm: new subtree 0x8b4210 liblognorm: addSampToTree 138 of 167 liblognorm: parsed literal: ' port ' liblognorm: buildPTree: begin at 0x8b4210, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 6, offs 0 liblognorm: addSampToTree 144 of 167 liblognorm: parsed field: 'src_port' liblognorm: got new subtree 0x8b4ac0 liblognorm: prev subtree 0x8b4210 liblognorm: new subtree 0x8b4ac0 liblognorm: addSampToTree 159 of 167 liblognorm: parsed field: '-' liblognorm: ERROR: invalid field type 'rest' liblognorm: read sample line: 'rule=:sshd%-:rest%' liblognorm: sample line to add: ':sshd%-:rest%' liblognorm: addSampToTree 0 of 59 liblognorm: parsed literal: '<' liblognorm: buildPTree: begin at 0x8b0060, offs 0 liblognorm: buildPTree: tree 0x8b0060, i 0, char '<' liblognorm: case 1.1 liblognorm: addSampToTree 1 of 59 liblognorm: parsed field: 'prio' liblognorm: got new subtree 0x8b5390 liblognorm: merging with tree 0x8b0df0 liblognorm: addSampToTree 14 of 59 liblognorm: parsed literal: '>' liblognorm: buildPTree: begin at 0x8b0df0, offs 0 liblognorm: buildPTree: tree 0x8b0df0, i 0, char '>' liblognorm: case 1.1 liblognorm: addSampToTree 15 of 59 liblognorm: parsed field: 'time' liblognorm: got new subtree 0x8b5c40 liblognorm: merging with tree 0x8b16a0 liblognorm: addSampToTree 34 of 59 liblognorm: parsed literal: ' ' liblognorm: buildPTree: begin at 0x8b16a0, offs 0 liblognorm: buildPTree: tree 0x8b16a0, i 0, char ' ' liblognorm: case 1.1 liblognorm: addSampToTree 35 of 59 liblognorm: parsed field: 'host' liblognorm: got new subtree 0x8b64f0 liblognorm: merging with tree 0x8b1f50 liblognorm: addSampToTree 46 of 59 liblognorm: parsed literal: ' sshd' liblognorm: buildPTree: begin at 0x8b1f50, offs 0 liblognorm: buildPTree: tree 0x8b1f50, i 0, char ' ' liblognorm: buildPTree: tree 0x8b1f50, i 1, char 's' liblognorm: buildPTree: tree 0x8b1f50, i 2, char 's' liblognorm: buildPTree: tree 0x8b1f50, i 3, char 'h' liblognorm: buildPTree: tree 0x8b1f50, i 4, char 'd' liblognorm: case 1.2 liblognorm: splitTree 0x8b1f50 at offs 5 liblognorm: setPrefix lenBuf 5, offs 0 liblognorm: splitTree new tree 0x8b6d40 lenPrefix=5, char ' ' liblognorm: splitTree new case two bb, offs=5, newlen 0 liblognorm: addSampToTree 51 of 59 liblognorm: parsed field: '-' liblognorm: ERROR: invalid field type 'rest' liblognorm: read sample line: 'annotate=passwd:+proc_name="passwd"' liblognorm: sample annotation to add: 'passwd:+proc_name="passwd"' liblognorm: read sample line: 'annotate=netbackup:+proc_name="netbackup"' liblognorm: sample annotation to add: 'netbackup:+proc_name="netbackup"' liblognorm: read sample line: 'annotate=httpd:+proc_name="httpd"' liblognorm: sample annotation to add: 'httpd:+proc_name="httpd"' liblognorm: read sample line: 'annotate=crond:+proc_name="crond"' liblognorm: sample annotation to add: 'crond:+proc_name="crond"' liblognorm: read sample line: 'annotate=atd:+proc_name="atd"' liblognorm: sample annotation to add: 'atd:+proc_name="atd"' liblognorm: read sample line: 'annotate=sudo:+proc_name="sudo"' liblognorm: sample annotation to add: 'sudo:+proc_name="sudo"' liblognorm: read sample line: 'annotate=su:+proc_name="su"' liblognorm: sample annotation to add: 'su:+proc_name="su"' liblognorm: read sample line: 'annotate=sshd:+proc_name="sshd"' liblognorm: sample annotation to add: 'sshd:+proc_name="sshd"' liblognorm: read sample line: 'annotate=dhcpd:+proc_name="dhcpd"' liblognorm: sample annotation to add: 'dhcpd:+proc_name="dhcpd"' liblognorm: read sample line: 'annotate=esx-vpxa:+proc_name="esx-vpxa"' liblognorm: sample annotation to add: 'esx-vpxa:+proc_name="esx-vpxa"' liblognorm: invalid tag field in annotation, line is 'annotate=esx-vpxa:+proc_name="esx-vpxa"' liblognorm: read sample line: 'annotate=vmware:+proc_name="vmware"' liblognorm: sample annotation to add: 'vmware:+proc_name="vmware"' liblognorm: read sample line: 'annotate=vmware_dcui:+proc_name="vmware-dcui"' liblognorm: sample annotation to add: 'vmware_dcui:+proc_name="vmware-dcui"' liblognorm: read sample line: 'annotate=vmware_vobd:+proc_name="vmware-vobd"' liblognorm: sample annotation to add: 'vmware_vobd:+proc_name="vmware-vobd"' liblognorm: read sample line: 'annotate=vmware_hostd:+proc_name="vmware-hostd"' liblognorm: sample annotation to add: 'vmware_hostd:+proc_name="vmware-hostd"' liblognorm: read sample line: 'annotate=vmware_vmauthd:+proc_name="vmware-vmauthd"' liblognorm: sample annotation to add: 'vmware_vmauthd:+proc_name="vmware-vmauthd"' liblognorm: read sample line: 'annotate=rsyslogd:+proc_name="rsyslogd"' liblognorm: sample annotation to add: 'rsyslogd:+proc_name="rsyslogd"' liblognorm: read sample line: 'annotate=autofail:+action="denied"' liblognorm: sample annotation to add: 'autofail:+action="denied"' liblognorm: read sample line: 'annotate=autosucc:+action="authorized"' liblognorm: sample annotation to add: 'autosucc:+action="authorized"' liblognorm: read sample line: 'annotate=authfail:+action="auth-fail"' liblognorm: sample annotation to add: 'authfail:+action="auth-fail"' liblognorm: read sample line: 'annotate=authsucc:+action="auth"' liblognorm: sample annotation to add: 'authsucc:+action="auth"' liblognorm: read sample line: 'annotate=sesopen:+action="ses-opened"' liblognorm: sample annotation to add: 'sesopen:+action="ses-opened"' liblognorm: read sample line: 'annotate=sesclose:+action="ses-closed"' liblognorm: sample annotation to add: 'sesclose:+action="ses-closed"' liblognorm: read sample line: 'annotate=invuser:+details="user invalid"' liblognorm: sample annotation to add: 'invuser:+details="user invalid"' liblognorm: read sample line: 'annotate=exec:+action="exec"' liblognorm: sample annotation to add: 'exec:+action="exec"' liblognorm: read sample line: 'annotate=pwch:+action="password change"' liblognorm: sample annotation to add: 'pwch:+action="password change"' liblognorm: read sample line: 'annotate=open:+action="conn-open"' liblognorm: sample annotation to add: 'open:+action="conn-open"' liblognorm: read sample line: 'annotate=close:+action="conn-close"' liblognorm: sample annotation to add: 'close:+action="conn-close"' liblognorm: read sample line: 'annotate=discon:+action="disconn"' liblognorm: sample annotation to add: 'discon:+action="disconn"' liblognorm: read sample line: 'annotate=discover:+action="discovery"' liblognorm: sample annotation to add: 'discover:+action="discovery"' liblognorm: read sample line: 'annotate=proterr:+action="prot-error"' liblognorm: sample annotation to add: 'proterr:+action="prot-error"' liblognorm: read sample line: 'annotate=hb:+action="heartbeat"' liblognorm: sample annotation to add: 'hb:+action="heartbeat"' number of tree nodes: 13 To normalize: '<86>Jun 1 12:37:41 felix sshd[23795]: Failed password for root from ::1 port 54849 ssh2' liblognorm: 0: prefix compare '<', '<' liblognorm: 1: prefix compare succeeded, still valid liblognorm: 1:trying parser for field 'prio': 0x7fcfc09bb4d0 liblognorm: potential hit, trying subtree liblognorm: 3: prefix compare '>', '>' liblognorm: 4: prefix compare succeeded, still valid liblognorm: 4:trying parser for field 'time': 0x7fcfc09baf70 liblognorm: potential hit, trying subtree liblognorm: 19: prefix compare ' ', ' ' liblognorm: 20: prefix compare succeeded, still valid liblognorm: 20:trying parser for field 'host': 0x7fcfc09ba790 liblognorm: potential hit, trying subtree liblognorm: 25: prefix compare ' ', ' ' liblognorm: 26: prefix compare 's', 's' liblognorm: 27: prefix compare 's', 's' liblognorm: 28: prefix compare 'h', 'h' liblognorm: 29: prefix compare 'd', 'd' liblognorm: 30: prefix compare succeeded, still valid liblognorm: 30 no field, trying subtree char '[': 0x8b1f50 liblognorm: 31: prefix compare succeeded, still valid liblognorm: 31:trying parser for field 'procid': 0x7fcfc09bb4d0 liblognorm: potential hit, trying subtree liblognorm: 36: prefix compare ']', ']' liblognorm: 37: prefix compare ':', ':' liblognorm: 38: prefix compare ' ', ' ' liblognorm: 39: prefix compare 'F', 'F' liblognorm: 40: prefix compare 'a', 'a' liblognorm: 41: prefix compare 'i', 'i' liblognorm: 42: prefix compare 'l', 'l' liblognorm: 43: prefix compare 'e', 'e' liblognorm: 44: prefix compare 'd', 'd' liblognorm: 45: prefix compare ' ', ' ' liblognorm: 46: prefix compare succeeded, still valid liblognorm: 46:trying parser for field 'user_authmethod': 0x7fcfc09ba790 liblognorm: potential hit, trying subtree liblognorm: 54: prefix compare ' ', ' ' liblognorm: 55: prefix compare 'f', 'f' liblognorm: 56: prefix compare 'o', 'o' liblognorm: 57: prefix compare 'r', 'r' liblognorm: 58: prefix compare ' ', ' ' liblognorm: 59: prefix compare succeeded, still valid liblognorm: 59:trying parser for field 'username': 0x7fcfc09ba790 liblognorm: potential hit, trying subtree liblognorm: 63: prefix compare ' ', ' ' liblognorm: 64: prefix compare 'f', 'f' liblognorm: 65: prefix compare 'r', 'r' liblognorm: 66: prefix compare 'o', 'o' liblognorm: 67: prefix compare 'm', 'm' liblognorm: 68: prefix compare ' ', ' ' liblognorm: 69: prefix compare succeeded, still valid liblognorm: 69:trying parser for field 'src_ip': 0x7fcfc09ba790 liblognorm: potential hit, trying subtree liblognorm: 72: prefix compare ' ', ' ' liblognorm: 73: prefix compare 'p', 'p' liblognorm: 74: prefix compare 'o', 'o' liblognorm: 75: prefix compare 'r', 'r' liblognorm: 76: prefix compare 't', 't' liblognorm: 77: prefix compare ' ', ' ' liblognorm: 78: prefix compare succeeded, still valid liblognorm: 78:trying parser for field 'src_port': 0x7fcfc09ba790 liblognorm: potential hit, trying subtree liblognorm: 83: prefix compare succeeded, still valid liblognorm: 83 no field, trying subtree char ' ': (nil) liblognorm: 83 returns 5 liblognorm: 78 nonmatch, backtracking required, left=5 liblognorm: 78 no field, trying subtree char '5': (nil) liblognorm: 78 returns 5 liblognorm: 69 nonmatch, backtracking required, left=5 liblognorm: 69 no field, trying subtree char ':': (nil) liblognorm: 69 returns 5 liblognorm: 59 nonmatch, backtracking required, left=5 liblognorm: 59 no field, trying subtree char 'r': (nil) liblognorm: 59 returns 5 liblognorm: 46 nonmatch, backtracking required, left=5 liblognorm: 46 no field, trying subtree char 'p': (nil) liblognorm: 46 returns 5 liblognorm: 31 nonmatch, backtracking required, left=5 liblognorm: 31 no field, trying subtree char '2': (nil) liblognorm: 31 returns 5 liblognorm: 30 returns 5 liblognorm: 20 nonmatch, backtracking required, left=5 liblognorm: 20 no field, trying subtree char 'f': (nil) liblognorm: 20 returns 5 liblognorm: 4 nonmatch, backtracking required, left=5 liblognorm: 4 no field, trying subtree char 'J': (nil) liblognorm: 4 returns 5 liblognorm: 1 nonmatch, backtracking required, left=5 liblognorm: 1 no field, trying subtree char '8': (nil) liblognorm: 1 returns 5 liblognorm: final result for normalizer: left 5, endNode 0x34352074726f7020 normalized: '[cee@115 originalmsg="<86>Jun 1 12:37:41 server sshd[23795\]: Failed password for root from ::1 port 54849 ssh2" unparsed-data=" ssh2"]' [cee@115 originalmsg="<86>Jun 1 12:37:41 server sshd[23795\]: Failed password for root from ::1 port 54849 ssh2" unparsed-data=" ssh2"] _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
