Yep, I guess it goes number (or ipv4 etc) > word > char-to (and char-sep) > rest.
The meta fields (such as tokenized, interpreted etc should have priority equal to the field actually used to match content internally. I guess we should have an default priority order, and allow user to change that). -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Apr 5, 2015 2:58 AM, "David Lang" <[email protected]> wrote: > On Sat, 4 Apr 2015, singh.janmejay wrote: > > As in current lognorm impl, field is always preferred to a literal. Rest >> is >> not really the problem, the relative order of fields and literals is. >> >> Is or fair to say that the rule with highest length is the best match? May >> be give higher weights to field, slightly lower to literals? >> > > not all field types are equal. > > if word and number both match, number is a better match > > rest is a last resort match > > There are probably other good examples that I'm not thinking of at the > moment > > I do think that longest match is the best match is going to be the right > way to go. > > David Lang > > -- >> Regards, >> Janmejay >> >> PS: Please blame the typos in this mail on my phone's uncivilized soft >> keyboard sporting it's not-so-smart-assist technology. >> >> On Apr 4, 2015 4:21 PM, "Rainer Gerhards" <[email protected]> >> wrote: >> >> Yeah, that's why I as always opposed against "rest" and similar >>> approaches. But there was so much pressure to have them, so I gave in. >>> Now we see that it really doesn't work out ;) >>> >>> The longer term solution (summer?) is what I am currently working on. >>> It'll solve these problems. >>> >>> Rainer >>> >>> 2015-04-03 5:51 GMT+02:00 David Lang <[email protected]>: >>> >>>> I'm getting back to looking at the cisco log parsing problem and I'm >>>> >>> seeing >>> >>>> odd behavior with the matching. I initially thought that it was tied to >>>> >>> the >>> >>>> descent stuff, but then testing just that subset and I'm seeing issues >>>> related to the 'rest' type. >>>> >>>> with the command: >>>> echo "Outside:6.65.60.79/53 (6.65.60.79/53)" >>>> >>> |/usr/lib/lognorm/lognormalizer >>> >>>> -r rules >>>> >>>> any of the following three lines will match if I comment out the other >>>> >>> two >>> >>>> >>>> rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number% >>>> >>> (%label2:char-to:)%) >>> >>>> >>>> rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number% >>>> (%label2:char-to:)%)%tail:rest% >>>> >>>> rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%%tail:rest% >>>> >>>> The later lines override the earlier ones. >>>> >>>> I would expect that the first line would be the one used as it's the >>>> best >>>> match. >>>> >>>> According to our earlier discussion, the second line should not match >>>> because there is nothing left to match after the ), but in this test it >>>> >>> does >>> >>>> and tail="" >>>> >>>> However, the real problem is that the last line is being preferred over >>>> >>> the >>> >>>> others, rest is not a last-chance match, it's being matched in >>>> >>> preference to >>> >>>> the ' (' literal text. it doesn't matter what the order is (including >>>> >>> when >>> >>>> this is used as a descent target where the order sometimes matters) >>>> >>>> the system is running liblognorm1 1.1.1-0adiscon1trusty1 amd64 >>>> >>>> David Lang >>>> >>>> To normalize: 'Outside:6.65.60.79/53 (6.65.60.79/53)' >>>> liblognorm: 0: prefix compare succeeded, still valid >>>> liblognorm: 0:trying parser for field 'iface': 0x7ffabbbd4870 >>>> liblognorm: parser returns 0, parsed 7 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 7: prefix compare ':', ':' >>>> liblognorm: 8: prefix compare succeeded, still valid >>>> liblognorm: 8:trying parser for field 'ip': 0x7ffabbbd6840 >>>> liblognorm: parser returns 0, parsed 10 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 18: prefix compare '/', '/' >>>> liblognorm: 19: prefix compare succeeded, still valid >>>> liblognorm: 19:trying parser for field 'port': 0x7ffabbbd4530 >>>> liblognorm: parser returns 0, parsed 2 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 21: prefix compare succeeded, still valid >>>> liblognorm: 21:trying parser for field 'tail': 0x7ffabbbd63a0 >>>> liblognorm: parser returns 0, parsed 16 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 37 returns 0 >>>> liblognorm: 21: parser matches at 21 >>>> liblognorm: 21 returns 0 >>>> liblognorm: 19: parser matches at 19 >>>> liblognorm: 19 returns 0 >>>> liblognorm: 8: parser matches at 8 >>>> liblognorm: 8 returns 0 >>>> liblognorm: 0: parser matches at 0 >>>> liblognorm: 0 returns 0 >>>> liblognorm: final result for normalizer: left 0, endNode 0x7ffabe37b210, >>>> isTerminal 1, tagbucket (nil) >>>> normalized: '[cee@115 tail=" (6.65.60.79/53)" port="53" ip="6.65.60.79" >>>> iface="Outside"]' >>>> [cee@115 tail=" (6.65.60.79/53)" port="53" ip="6.65.60.79" >>>> >>> iface="Outside"] >>> >>>> >>>> comment out the last line >>>> >>>> To normalize: 'Outside:6.65.60.79/53 (6.65.60.79/53)' >>>> liblognorm: 0: prefix compare succeeded, still valid >>>> liblognorm: 0:trying parser for field 'iface': 0x7f41f814e870 >>>> liblognorm: parser returns 0, parsed 7 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 7: prefix compare ':', ':' >>>> liblognorm: 8: prefix compare succeeded, still valid >>>> liblognorm: 8:trying parser for field 'ip': 0x7f41f8150840 >>>> liblognorm: parser returns 0, parsed 10 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 18: prefix compare '/', '/' >>>> liblognorm: 19: prefix compare succeeded, still valid >>>> liblognorm: 19:trying parser for field 'port': 0x7f41f814e530 >>>> liblognorm: parser returns 0, parsed 2 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 21: prefix compare ' ', ' ' >>>> liblognorm: 22: prefix compare '(', '(' >>>> liblognorm: 23: prefix compare succeeded, still valid >>>> liblognorm: 23:trying parser for field 'label2': 0x7f41f814e870 >>>> liblognorm: parser returns 0, parsed 13 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 36: prefix compare ')', ')' >>>> liblognorm: 37: prefix compare succeeded, still valid >>>> liblognorm: 37:trying parser for field 'tail': 0x7f41f81503a0 >>>> liblognorm: parser returns 0, parsed 0 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 37 returns 0 >>>> liblognorm: 37: parser matches at 37 >>>> liblognorm: 37 returns 0 >>>> liblognorm: 23: parser matches at 23 >>>> liblognorm: 23 returns 0 >>>> liblognorm: 19: parser matches at 19 >>>> liblognorm: 19 returns 0 >>>> liblognorm: 8: parser matches at 8 >>>> liblognorm: 8 returns 0 >>>> liblognorm: 0: parser matches at 0 >>>> liblognorm: 0 returns 0 >>>> liblognorm: final result for normalizer: left 0, endNode 0x7f41f97df170, >>>> isTerminal 1, tagbucket (nil) >>>> normalized: '[cee@115 tail="" label2="6.65.60.79/53" port="53" >>>> ip="6.65.60.79" iface="Outside"]' >>>> [cee@115 tail="" label2="6.65.60.79/53" port="53" ip="6.65.60.79" >>>> iface="Outside"] >>>> >>>> and finally just the first line >>>> >>>> To normalize: 'Outside:6.65.60.79/53 (6.65.60.79/53)' >>>> liblognorm: 0: prefix compare succeeded, still valid >>>> liblognorm: 0:trying parser for field 'iface': 0x7fb1029a9870 >>>> liblognorm: parser returns 0, parsed 7 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 7: prefix compare ':', ':' >>>> liblognorm: 8: prefix compare succeeded, still valid >>>> liblognorm: 8:trying parser for field 'ip': 0x7fb1029ab840 >>>> liblognorm: parser returns 0, parsed 10 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 18: prefix compare '/', '/' >>>> liblognorm: 19: prefix compare succeeded, still valid >>>> liblognorm: 19:trying parser for field 'port': 0x7fb1029a9530 >>>> liblognorm: parser returns 0, parsed 2 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 21: prefix compare ' ', ' ' >>>> liblognorm: 22: prefix compare '(', '(' >>>> liblognorm: 23: prefix compare succeeded, still valid >>>> liblognorm: 23:trying parser for field 'label2': 0x7fb1029a9870 >>>> liblognorm: parser returns 0, parsed 13 >>>> liblognorm: potential hit, trying subtree >>>> liblognorm: 36: prefix compare ')', ')' >>>> liblognorm: 37: prefix compare succeeded, still valid >>>> liblognorm: 37 returns 0 >>>> liblognorm: 23: parser matches at 23 >>>> liblognorm: 23 returns 0 >>>> liblognorm: 19: parser matches at 19 >>>> liblognorm: 19 returns 0 >>>> liblognorm: 8: parser matches at 8 >>>> liblognorm: 8 returns 0 >>>> liblognorm: 0: parser matches at 0 >>>> liblognorm: 0 returns 0 >>>> liblognorm: final result for normalizer: left 0, endNode 0x7fb1033027f0, >>>> isTerminal 1, tagbucket (nil) >>>> normalized: '[cee@115 label2="6.65.60.79/53" port="53" ip="6.65.60.79" >>>> iface="Outside"]' >>>> [cee@115 label2="6.65.60.79/53" port="53" ip="6.65.60.79" >>>> >>> iface="Outside"] >>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> >>> of >>> >>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T >>>> LIKE THAT. >>>> >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
