Hi David, I totally agree with you. This is something most people would appreciate. I bump into these issues with customers quite often. Not sure if the parsers for various formats would be obsoleted by this generic parser, as most are not that simple and have many variations.
On the other hand, the change I am proposing is a one liner or a parser that is very similar to the RFC5424. Both are easy to implement. Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Tue, Nov 24, 2015 at 11:45 AM, David Lang <[email protected]> wrote: > what we really need is the ability to define a user-defined type that > extracts the various datetime items and returns a timestamp value. Allowing > the admin to specify individual datetime type items in a pattern. > > Ideally this would use the 'standard' names that date can use to format > it's output. > > This is more work than just adding one more date parser type, but with all > the garbage that can end up in logs and the different ways that times can > be formatted, we can't create a parser for every possible type, and there's > a lot of value in being able to extract the day here, month there, time > somewhere else, timezone, and having the result be something you can treat > like you do $timestamp with it's output formatting options. > > David Lang > > > On Tue, 24 Nov 2015, Rainer Gerhards wrote: > > Date: Tue, 24 Nov 2015 09:54:10 +0100 >> From: Rainer Gerhards <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] Time Format >> >> >> ahhh, I was so focussed on the RFC5424 parser. If it's lognorm, I >> suggest to add a feature request tracker. I am right now cleaning up >> things, but starting January I can begin to larger implementation. And >> if it turns out to be small, I may be able to sneak it in. But let's >> file a bug tracker with all relevant info, that makes it much more >> probably this will materialize. >> >> Rainer >> >> 2015-11-24 9:47 GMT+01:00 Radu Gheorghe <[email protected]>: >> >>> Hello, >>> >>> I think the actual need for this functionality would be outside >>> RFC5424. Or RFC3164 for that matter. >>> >>> It sounds like Vicks (and also Ciprian and I) would need it as a >>> function of mmnormalize/liblognorm so that we can parse logs from >>> files. This different format in the Email is something I often see in >>> Java logs. >>> >>> The more general use-case would be to parse all kinds of date formats >>> (mysql, apache, whatever - it seems like there's a billion of them). >>> Currently the only option I'm aware of is to hack around with parsing >>> different parts of the date as a string and stitching it in the >>> template. All very ugly. >>> >>> @Ciprian and Vicks: please let me know if I misinterpreted what you >>> wanted. What I describe here is what I would find useful. >>> >>> Best regards, >>> Radu >>> >>> P.S. Now that I think of it, it wouldn't be only useful for parsing >>> logs from files. It could be that some apps just send logs over TCP >>> (say, newline-delimited) that don't comply to either of the syslog >>> RFCs. And then we could use mmnormalize to parse them. Goes into the >>> direction of "rsyslog is not only for syslog". >>> -- >>> Performance Monitoring * Log Analytics * Search Analytics >>> Solr & Elasticsearch Support * http://sematext.com/ >>> >>> >>> On Tue, Nov 24, 2015 at 10:37 AM, Rainer Gerhards >>> <[email protected]> wrote: >>> >>>> 2015-11-24 9:18 GMT+01:00 David Lang <[email protected]>: >>>> >>>>> On Tue, 24 Nov 2015, Ciprian Hacman wrote: >>>>> >>>>> I was actually thinking of creating a PR for accepting " " instead of >>>>>> "T" >>>>>> between date and time. >>>>>> @Rainer: Would it be ok? >>>>>> >>>>> >>>>> >>>>> my reaction is that it depends on how paranoid the rest of the code >>>>> is. Is >>>>> there any chance that this will cause it to misinterpret something >>>>> else as a >>>>> match? >>>>> >>>> >>>> No, but the current stance of the IETF is "if it's malformed, than >>>> it's dangerous". I think that paradigm is correct to follow these >>>> days. An option would work, but the default should be to comply with >>>> RFC rules. >>>> >>>> Rainer >>>> >>>>> >>>>> David Lang >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad of >>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T >>>>> LIKE THAT. >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> you DON'T LIKE THAT. >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
