2015-04-05 5:30 GMT+02:00 singh.janmejay <[email protected]>: > Yep, I guess it goes number (or ipv4 etc) > word > char-to (and char-sep) > > rest. > > The meta fields (such as tokenized, interpreted etc should have priority > equal to the field actually used to match content internally. > > I guess we should have an default priority order, and allow user to change > that).
I have just created a fix for the rest problem: https://github.com/rgerhards/liblognorm/tree/fix-rest It would be great if someone could try it out. It's a bit quick and dirty, but I think most of that code will be changed anyway as I am going forward with my new lognorm developments. That's also the reason why I would not yet like to include parser priorities right now. Rainer > > -- > Regards, > Janmejay > > PS: Please blame the typos in this mail on my phone's uncivilized soft > keyboard sporting it's not-so-smart-assist technology. > > On Apr 5, 2015 2:58 AM, "David Lang" <[email protected]> wrote: > >> On Sat, 4 Apr 2015, singh.janmejay wrote: >> >> As in current lognorm impl, field is always preferred to a literal. Rest >>> is >>> not really the problem, the relative order of fields and literals is. >>> >>> Is or fair to say that the rule with highest length is the best match? May >>> be give higher weights to field, slightly lower to literals? >>> >> >> not all field types are equal. >> >> if word and number both match, number is a better match >> >> rest is a last resort match >> >> There are probably other good examples that I'm not thinking of at the >> moment >> >> I do think that longest match is the best match is going to be the right >> way to go. >> >> David Lang >> >> -- >>> Regards, >>> Janmejay >>> >>> PS: Please blame the typos in this mail on my phone's uncivilized soft >>> keyboard sporting it's not-so-smart-assist technology. >>> >>> On Apr 4, 2015 4:21 PM, "Rainer Gerhards" <[email protected]> >>> wrote: >>> >>> Yeah, that's why I as always opposed against "rest" and similar >>>> approaches. But there was so much pressure to have them, so I gave in. >>>> Now we see that it really doesn't work out ;) >>>> >>>> The longer term solution (summer?) is what I am currently working on. >>>> It'll solve these problems. >>>> >>>> Rainer >>>> >>>> 2015-04-03 5:51 GMT+02:00 David Lang <[email protected]>: >>>> >>>>> I'm getting back to looking at the cisco log parsing problem and I'm >>>>> >>>> seeing >>>> >>>>> odd behavior with the matching. I initially thought that it was tied to >>>>> >>>> the >>>> >>>>> descent stuff, but then testing just that subset and I'm seeing issues >>>>> related to the 'rest' type. >>>>> >>>>> with the command: >>>>> echo "Outside:6.65.60.79/53 (6.65.60.79/53)" >>>>> >>>> |/usr/lib/lognorm/lognormalizer >>>> >>>>> -r rules >>>>> >>>>> any of the following three lines will match if I comment out the other >>>>> >>>> two >>>> >>>>> >>>>> rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number% >>>>> >>>> (%label2:char-to:)%) >>>> >>>>> >>>>> rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number% >>>>> (%label2:char-to:)%)%tail:rest% >>>>> >>>>> rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%%tail:rest% >>>>> >>>>> The later lines override the earlier ones. >>>>> >>>>> I would expect that the first line would be the one used as it's the >>>>> best >>>>> match. >>>>> >>>>> According to our earlier discussion, the second line should not match >>>>> because there is nothing left to match after the ), but in this test it >>>>> >>>> does >>>> >>>>> and tail="" >>>>> >>>>> However, the real problem is that the last line is being preferred over >>>>> >>>> the >>>> >>>>> others, rest is not a last-chance match, it's being matched in >>>>> >>>> preference to >>>> >>>>> the ' (' literal text. it doesn't matter what the order is (including >>>>> >>>> when >>>> >>>>> this is used as a descent target where the order sometimes matters) >>>>> >>>>> the system is running liblognorm1 1.1.1-0adiscon1trusty1 amd64 >>>>> >>>>> David Lang >>>>> >>>>> To normalize: 'Outside:6.65.60.79/53 (6.65.60.79/53)' >>>>> liblognorm: 0: prefix compare succeeded, still valid >>>>> liblognorm: 0:trying parser for field 'iface': 0x7ffabbbd4870 >>>>> liblognorm: parser returns 0, parsed 7 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 7: prefix compare ':', ':' >>>>> liblognorm: 8: prefix compare succeeded, still valid >>>>> liblognorm: 8:trying parser for field 'ip': 0x7ffabbbd6840 >>>>> liblognorm: parser returns 0, parsed 10 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 18: prefix compare '/', '/' >>>>> liblognorm: 19: prefix compare succeeded, still valid >>>>> liblognorm: 19:trying parser for field 'port': 0x7ffabbbd4530 >>>>> liblognorm: parser returns 0, parsed 2 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 21: prefix compare succeeded, still valid >>>>> liblognorm: 21:trying parser for field 'tail': 0x7ffabbbd63a0 >>>>> liblognorm: parser returns 0, parsed 16 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 37 returns 0 >>>>> liblognorm: 21: parser matches at 21 >>>>> liblognorm: 21 returns 0 >>>>> liblognorm: 19: parser matches at 19 >>>>> liblognorm: 19 returns 0 >>>>> liblognorm: 8: parser matches at 8 >>>>> liblognorm: 8 returns 0 >>>>> liblognorm: 0: parser matches at 0 >>>>> liblognorm: 0 returns 0 >>>>> liblognorm: final result for normalizer: left 0, endNode 0x7ffabe37b210, >>>>> isTerminal 1, tagbucket (nil) >>>>> normalized: '[cee@115 tail=" (6.65.60.79/53)" port="53" ip="6.65.60.79" >>>>> iface="Outside"]' >>>>> [cee@115 tail=" (6.65.60.79/53)" port="53" ip="6.65.60.79" >>>>> >>>> iface="Outside"] >>>> >>>>> >>>>> comment out the last line >>>>> >>>>> To normalize: 'Outside:6.65.60.79/53 (6.65.60.79/53)' >>>>> liblognorm: 0: prefix compare succeeded, still valid >>>>> liblognorm: 0:trying parser for field 'iface': 0x7f41f814e870 >>>>> liblognorm: parser returns 0, parsed 7 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 7: prefix compare ':', ':' >>>>> liblognorm: 8: prefix compare succeeded, still valid >>>>> liblognorm: 8:trying parser for field 'ip': 0x7f41f8150840 >>>>> liblognorm: parser returns 0, parsed 10 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 18: prefix compare '/', '/' >>>>> liblognorm: 19: prefix compare succeeded, still valid >>>>> liblognorm: 19:trying parser for field 'port': 0x7f41f814e530 >>>>> liblognorm: parser returns 0, parsed 2 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 21: prefix compare ' ', ' ' >>>>> liblognorm: 22: prefix compare '(', '(' >>>>> liblognorm: 23: prefix compare succeeded, still valid >>>>> liblognorm: 23:trying parser for field 'label2': 0x7f41f814e870 >>>>> liblognorm: parser returns 0, parsed 13 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 36: prefix compare ')', ')' >>>>> liblognorm: 37: prefix compare succeeded, still valid >>>>> liblognorm: 37:trying parser for field 'tail': 0x7f41f81503a0 >>>>> liblognorm: parser returns 0, parsed 0 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 37 returns 0 >>>>> liblognorm: 37: parser matches at 37 >>>>> liblognorm: 37 returns 0 >>>>> liblognorm: 23: parser matches at 23 >>>>> liblognorm: 23 returns 0 >>>>> liblognorm: 19: parser matches at 19 >>>>> liblognorm: 19 returns 0 >>>>> liblognorm: 8: parser matches at 8 >>>>> liblognorm: 8 returns 0 >>>>> liblognorm: 0: parser matches at 0 >>>>> liblognorm: 0 returns 0 >>>>> liblognorm: final result for normalizer: left 0, endNode 0x7f41f97df170, >>>>> isTerminal 1, tagbucket (nil) >>>>> normalized: '[cee@115 tail="" label2="6.65.60.79/53" port="53" >>>>> ip="6.65.60.79" iface="Outside"]' >>>>> [cee@115 tail="" label2="6.65.60.79/53" port="53" ip="6.65.60.79" >>>>> iface="Outside"] >>>>> >>>>> and finally just the first line >>>>> >>>>> To normalize: 'Outside:6.65.60.79/53 (6.65.60.79/53)' >>>>> liblognorm: 0: prefix compare succeeded, still valid >>>>> liblognorm: 0:trying parser for field 'iface': 0x7fb1029a9870 >>>>> liblognorm: parser returns 0, parsed 7 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 7: prefix compare ':', ':' >>>>> liblognorm: 8: prefix compare succeeded, still valid >>>>> liblognorm: 8:trying parser for field 'ip': 0x7fb1029ab840 >>>>> liblognorm: parser returns 0, parsed 10 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 18: prefix compare '/', '/' >>>>> liblognorm: 19: prefix compare succeeded, still valid >>>>> liblognorm: 19:trying parser for field 'port': 0x7fb1029a9530 >>>>> liblognorm: parser returns 0, parsed 2 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 21: prefix compare ' ', ' ' >>>>> liblognorm: 22: prefix compare '(', '(' >>>>> liblognorm: 23: prefix compare succeeded, still valid >>>>> liblognorm: 23:trying parser for field 'label2': 0x7fb1029a9870 >>>>> liblognorm: parser returns 0, parsed 13 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 36: prefix compare ')', ')' >>>>> liblognorm: 37: prefix compare succeeded, still valid >>>>> liblognorm: 37 returns 0 >>>>> liblognorm: 23: parser matches at 23 >>>>> liblognorm: 23 returns 0 >>>>> liblognorm: 19: parser matches at 19 >>>>> liblognorm: 19 returns 0 >>>>> liblognorm: 8: parser matches at 8 >>>>> liblognorm: 8 returns 0 >>>>> liblognorm: 0: parser matches at 0 >>>>> liblognorm: 0 returns 0 >>>>> liblognorm: final result for normalizer: left 0, endNode 0x7fb1033027f0, >>>>> isTerminal 1, tagbucket (nil) >>>>> normalized: '[cee@115 label2="6.65.60.79/53" port="53" ip="6.65.60.79" >>>>> iface="Outside"]' >>>>> [cee@115 label2="6.65.60.79/53" port="53" ip="6.65.60.79" >>>>> >>>> iface="Outside"] >>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>>> >>>> of >>>> >>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T >>>>> LIKE THAT. >>>>> >>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
