I like the nullmarker idea a lot, since that's one of the most common issue. Also, it solves it pretty efficiently. I think it needs to be in the rulebase, or liblognorm is tied to being only a part of rsyslog. Chris
On Tue Jan 27 2015 at 10:27:42 PM singh.janmejay <[email protected]> wrote: > I see what you are thinking of, but somethings that may be worth thinking > about before we decide: > > - Does it make sense for users to pack unrelated samples in the same > rulebase? > > There are 3 problems with this: > * The tree will become large, and back-tracking several unrelated > branches will be wasteful (a condition in ruleset which calls the action > will be much more efficient assuming tests is not very complex) > > * The rulebase will be composed of several unrelated rules, making it > harder to read > > * Multiple parse-trees may have to be maintained in order to satisfy > all combinations of nullMarker (eg. a non-leaf field, marked for > null-handling in one sample, but not marked for it in the other) (so > matching will become O(n) in number of combinations). So it is some > dev-work and little bit of perf-overhead. > > - The alternative is to set nullMarker at top level in a rulebase (instead > of being able to change it for every sample). > > But then the flexibility is slightly lowered. > > - If we go with action level param, its useful in cases where one has > standard access-log format but load-balancer level always have some fields > (say upstream latency or upstream-ip) which app-layer access logs will not > have. > > This can use the same rulebase with nullMarker in one case, and without > it in another. > > Thoughts? > > On Wed, Jan 28, 2015 at 11:13 AM, David Lang <[email protected]> wrote: > > > I'm thinking that it needs to only apply to part of a ruleset. I can't > see > > why you would use the same rulebase with different values overall, but I > > can easily see a rulebase that covers more than one type of logs needing > > different values for the different types of logs. > > > > remember that liblognorm is most effictive if it has one ruleset to cover > > everything you are looking at rather than doing other conditionals and > then > > picking which rulset to use. > > > > David Lang > > > > > > On Wed, 28 Jan 2015, singh.janmejay wrote: > > > > I think action parameter is the most flexible place to have it at. > Because > >> same rulebase can be used with different values. > >> > >> Either module or rulebase level param will be less flexible compared to > >> this. > >> > >> -- > >> Regards, > >> Janmejay > >> > >> PS: Please blame the typos in this mail on my phone's uncivilized soft > >> keyboard sporting it's not-so-smart-assist technology. > >> > >> On Jan 28, 2015 10:48 AM, "David Lang" <[email protected]> wrote: > >> > >> On Wed, 28 Jan 2015, singh.janmejay wrote: > >>> > >>> Ok, one way I can think of doing it: expose a parameter at > action/module > >>> > >>>> level which turns on defaulting and picks a default string. > >>>> > >>>> Eg. > >>>> > >>>> action(type="mmnormalize " nullMarker="-") > >>>> > >>>> Where nullMarker is a string (not a char). > >>>> > >>>> Whenever a "-" is encountered and a field is expected, it should skip > >>>> the > >>>> key(the key will not be present at all) and continue matching next > token > >>>> onwards. > >>>> > >>>> Thoughts? > >>>> > >>>> > >>> This needs to be something in the liblognorm config, not in rsyslog. > >>> different types of logs would have different nullMarker strings. > >>> > >>> with that adjustment, I think it's a good idea. > >>> > >>> David Lang > >>> > >>> -- > >>> > >>>> Regards, > >>>> Janmejay > >>>> > >>>> PS: Please blame the typos in this mail on my phone's uncivilized soft > >>>> keyboard sporting it's not-so-smart-assist technology. > >>>> > >>>> On Jan 28, 2015 6:38 AM, "David Lang" <[email protected]> wrote: > >>>> > >>>> On Wed, 28 Jan 2015, singh.janmejay wrote: > >>>> > >>>>> > >>>>> May be it'll be useful to discuss what you want to achieve with such > >>>>> > >>>>> representations of sample. I mean if possible, take a few samples > from > >>>>>> your > >>>>>> existing rulebase which you think highlight the problem(s) you are > >>>>>> facing. > >>>>>> > >>>>>> > >>>>>> I think the example is the Apache logs, where Apache either puts a > >>>>> value, > >>>>> or it puts a placeholder '-' > >>>>> > >>>>> if you want to capture a specific type (number or ip address for > >>>>> example), > >>>>> you won't match a log entry that has a - in that field. > >>>>> > >>>>> If there are only a couple fields that are like this, you can list > all > >>>>> the > >>>>> combinations in the ruleset, but if you have a lot of fields like > this, > >>>>> the > >>>>> combinatorial explosion would make for a LOT of rules. > >>>>> > >>>>> So I don't think he really needs a generic 'or' allowing any types to > >>>>> be > >>>>> combined as much as a way to say "this field could be this type or > this > >>>>> constant" > >>>>> > >>>>> David Lang > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com/professional-services/ > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>> myriad > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>> DON'T LIKE THAT. > >>>>> > >>>>> _______________________________________________ > >>>>> > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>>> _______________________________________________ > >>>> > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >>> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > >> _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > > > > -- > Regards, > Janmejay > http://codehunk.wordpress.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
