I see what you are thinking of, but somethings that may be worth thinking
about before we decide:
- Does it make sense for users to pack unrelated samples in the same
rulebase?
There are 3 problems with this:
* The tree will become large, and back-tracking several unrelated
branches will be wasteful (a condition in ruleset which calls the action
will be much more efficient assuming tests is not very complex)
* The rulebase will be composed of several unrelated rules, making it
harder to read
* Multiple parse-trees may have to be maintained in order to satisfy
all combinations of nullMarker (eg. a non-leaf field, marked for
null-handling in one sample, but not marked for it in the other) (so
matching will become O(n) in number of combinations). So it is some
dev-work and little bit of perf-overhead.
- The alternative is to set nullMarker at top level in a rulebase (instead
of being able to change it for every sample).
But then the flexibility is slightly lowered.
- If we go with action level param, its useful in cases where one has
standard access-log format but load-balancer level always have some fields
(say upstream latency or upstream-ip) which app-layer access logs will not
have.
This can use the same rulebase with nullMarker in one case, and without
it in another.
Thoughts?
On Wed, Jan 28, 2015 at 11:13 AM, David Lang <[email protected]> wrote:
> I'm thinking that it needs to only apply to part of a ruleset. I can't see
> why you would use the same rulebase with different values overall, but I
> can easily see a rulebase that covers more than one type of logs needing
> different values for the different types of logs.
>
> remember that liblognorm is most effictive if it has one ruleset to cover
> everything you are looking at rather than doing other conditionals and then
> picking which rulset to use.
>
> David Lang
>
>
> On Wed, 28 Jan 2015, singh.janmejay wrote:
>
> I think action parameter is the most flexible place to have it at. Because
>> same rulebase can be used with different values.
>>
>> Either module or rulebase level param will be less flexible compared to
>> this.
>>
>> --
>> Regards,
>> Janmejay
>>
>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>>
>> On Jan 28, 2015 10:48 AM, "David Lang" <[email protected]> wrote:
>>
>> On Wed, 28 Jan 2015, singh.janmejay wrote:
>>>
>>> Ok, one way I can think of doing it: expose a parameter at action/module
>>>
>>>> level which turns on defaulting and picks a default string.
>>>>
>>>> Eg.
>>>>
>>>> action(type="mmnormalize " nullMarker="-")
>>>>
>>>> Where nullMarker is a string (not a char).
>>>>
>>>> Whenever a "-" is encountered and a field is expected, it should skip
>>>> the
>>>> key(the key will not be present at all) and continue matching next token
>>>> onwards.
>>>>
>>>> Thoughts?
>>>>
>>>>
>>> This needs to be something in the liblognorm config, not in rsyslog.
>>> different types of logs would have different nullMarker strings.
>>>
>>> with that adjustment, I think it's a good idea.
>>>
>>> David Lang
>>>
>>> --
>>>
>>>> Regards,
>>>> Janmejay
>>>>
>>>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>>>> keyboard sporting it's not-so-smart-assist technology.
>>>>
>>>> On Jan 28, 2015 6:38 AM, "David Lang" <[email protected]> wrote:
>>>>
>>>> On Wed, 28 Jan 2015, singh.janmejay wrote:
>>>>
>>>>>
>>>>> May be it'll be useful to discuss what you want to achieve with such
>>>>>
>>>>> representations of sample. I mean if possible, take a few samples from
>>>>>> your
>>>>>> existing rulebase which you think highlight the problem(s) you are
>>>>>> facing.
>>>>>>
>>>>>>
>>>>>> I think the example is the Apache logs, where Apache either puts a
>>>>> value,
>>>>> or it puts a placeholder '-'
>>>>>
>>>>> if you want to capture a specific type (number or ip address for
>>>>> example),
>>>>> you won't match a log entry that has a - in that field.
>>>>>
>>>>> If there are only a couple fields that are like this, you can list all
>>>>> the
>>>>> combinations in the ruleset, but if you have a lot of fields like this,
>>>>> the
>>>>> combinatorial explosion would make for a LOT of rules.
>>>>>
>>>>> So I don't think he really needs a generic 'or' allowing any types to
>>>>> be
>>>>> combined as much as a way to say "this field could be this type or this
>>>>> constant"
>>>>>
>>>>> David Lang
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>>
>>>>> _______________________________________________
>>>>>
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>>> _______________________________________________
>>>>
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
--
Regards,
Janmejay
http://codehunk.wordpress.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
