2015-11-24 11:36 GMT+01:00 Ciprian Hacman <[email protected]>: > Hi David, > > I totally agree with you. This is something most people would appreciate. I > bump into these issues with customers quite often. > Not sure if the parsers for various formats would be obsoleted by this > generic parser, as most are not that simple and have many variations. > > On the other hand, the change I am proposing is a one liner or a parser > that is very similar to the RFC5424. Both are easy to implement.
I have no time to look into it right at the moment, but shouldn't that be something we can do with a custom data type (definition)? Anyhow, I'll add a note to the github tracker so that we can see when we go there. In general, I try to limit the core parsers, as in v2 we now really have much better capabilities. Rainer > > Ciprian > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > On Tue, Nov 24, 2015 at 11:45 AM, David Lang <[email protected]> wrote: > >> what we really need is the ability to define a user-defined type that >> extracts the various datetime items and returns a timestamp value. Allowing >> the admin to specify individual datetime type items in a pattern. >> >> Ideally this would use the 'standard' names that date can use to format >> it's output. >> >> This is more work than just adding one more date parser type, but with all >> the garbage that can end up in logs and the different ways that times can >> be formatted, we can't create a parser for every possible type, and there's >> a lot of value in being able to extract the day here, month there, time >> somewhere else, timezone, and having the result be something you can treat >> like you do $timestamp with it's output formatting options. >> >> David Lang >> >> >> On Tue, 24 Nov 2015, Rainer Gerhards wrote: >> >> Date: Tue, 24 Nov 2015 09:54:10 +0100 >>> From: Rainer Gerhards <[email protected]> >>> Reply-To: rsyslog-users <[email protected]> >>> To: rsyslog-users <[email protected]> >>> Subject: Re: [rsyslog] Time Format >>> >>> >>> ahhh, I was so focussed on the RFC5424 parser. If it's lognorm, I >>> suggest to add a feature request tracker. I am right now cleaning up >>> things, but starting January I can begin to larger implementation. And >>> if it turns out to be small, I may be able to sneak it in. But let's >>> file a bug tracker with all relevant info, that makes it much more >>> probably this will materialize. >>> >>> Rainer >>> >>> 2015-11-24 9:47 GMT+01:00 Radu Gheorghe <[email protected]>: >>> >>>> Hello, >>>> >>>> I think the actual need for this functionality would be outside >>>> RFC5424. Or RFC3164 for that matter. >>>> >>>> It sounds like Vicks (and also Ciprian and I) would need it as a >>>> function of mmnormalize/liblognorm so that we can parse logs from >>>> files. This different format in the Email is something I often see in >>>> Java logs. >>>> >>>> The more general use-case would be to parse all kinds of date formats >>>> (mysql, apache, whatever - it seems like there's a billion of them). >>>> Currently the only option I'm aware of is to hack around with parsing >>>> different parts of the date as a string and stitching it in the >>>> template. All very ugly. >>>> >>>> @Ciprian and Vicks: please let me know if I misinterpreted what you >>>> wanted. What I describe here is what I would find useful. >>>> >>>> Best regards, >>>> Radu >>>> >>>> P.S. Now that I think of it, it wouldn't be only useful for parsing >>>> logs from files. It could be that some apps just send logs over TCP >>>> (say, newline-delimited) that don't comply to either of the syslog >>>> RFCs. And then we could use mmnormalize to parse them. Goes into the >>>> direction of "rsyslog is not only for syslog". >>>> -- >>>> Performance Monitoring * Log Analytics * Search Analytics >>>> Solr & Elasticsearch Support * http://sematext.com/ >>>> >>>> >>>> On Tue, Nov 24, 2015 at 10:37 AM, Rainer Gerhards >>>> <[email protected]> wrote: >>>> >>>>> 2015-11-24 9:18 GMT+01:00 David Lang <[email protected]>: >>>>> >>>>>> On Tue, 24 Nov 2015, Ciprian Hacman wrote: >>>>>> >>>>>> I was actually thinking of creating a PR for accepting " " instead of >>>>>>> "T" >>>>>>> between date and time. >>>>>>> @Rainer: Would it be ok? >>>>>>> >>>>>> >>>>>> >>>>>> my reaction is that it depends on how paranoid the rest of the code >>>>>> is. Is >>>>>> there any chance that this will cause it to misinterpret something >>>>>> else as a >>>>>> match? >>>>>> >>>>> >>>>> No, but the current stance of the IETF is "if it's malformed, than >>>>> it's dangerous". I think that paradigm is correct to follow these >>>>> days. An option would work, but the default should be to comply with >>>>> RFC rules. >>>>> >>>>> Rainer >>>>> >>>>>> >>>>>> David Lang >>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad of >>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T >>>>>> LIKE THAT. >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>> you DON'T LIKE THAT. >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
