Hi, Ideally, IMHO, I see this working like this: - new custom date format parser for liblognorm that can use patterns similar to this: http://docs.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html - new message modification module that can parse any custom date format, and store it in timereported or in another variable in an internal format - new template property dateformat that can output from the above internal format to any custom date format (if you want to get just the year or the day for example)
Regarding the current issue, the ISO8601 format "2015-11-14 17:10:26,589" as described here <http://joda-time.sourceforge.net/api-release/org/joda/time/format/ISODateTimeFormat.html#dateTimeParser()> is very used in Java, with slight variations. At the moment, to extract it, I use this liblognorm pattern "%date:date-iso% %time:time-24hr%,%msec:number%". After extracting it, I have to assemble it into a single variable to be able to use it in a template: "set $!timestamp = $!date & "T" & $!time & "." & $!msec & "Z";". Seems that user defined types <https://github.com/rsyslog/liblognorm/blob/master/doc/configuration.rst#user-defined-types> might help here, but regular users might not find this feature. Given the wide use of ISO8601, I think a special parser for it would be nice. As David proposed, we can have a simple timestamp field type with multiple formats (rfcXXX, isoXXX, custom, ...). Most people that contact us, get lost somewhere at extracting/parsing/formating dates in this simple format. Sometimes they can change the date format to something that can be extracted using the RFC5424 parser, but most of the time they can't or they want to keep their log files more readable and have to do something similar to complicate the Rsyslog config to ship their logs. Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Nov 25, 2015 at 10:25 AM, Rainer Gerhards <[email protected]> wrote: > 2015-11-24 23:55 GMT+01:00 David Lang <[email protected]>: > > On Tue, 24 Nov 2015, Rainer Gerhards wrote: > > > >> 2015-11-24 11:36 GMT+01:00 Ciprian Hacman <[email protected] > >: > >>> > >>> Hi David, > >>> > >>> I totally agree with you. This is something most people would > appreciate. > >>> I > >>> bump into these issues with customers quite often. > >>> Not sure if the parsers for various formats would be obsoleted by this > >>> generic parser, as most are not that simple and have many variations. > >>> > >>> On the other hand, the change I am proposing is a one liner or a parser > >>> that is very similar to the RFC5424. Both are easy to implement. > >> > >> > >> I have no time to look into it right at the moment, but shouldn't that > >> be something we can do with a custom data type (definition)? > > > > > > not currently. > > > > I can create a custom data type '@mytimestamp' that contains a month, > day, > > year, hour, min, sec, fractsec extracted from a bizzar log line, but the > > only way for me to turn this into a timestamp is to do an exec_template > to > > format a variable based on these variables, and then run another > mmnormalize > > against that resulting string to get a timestamp. > > > > the goal is to be able to take arbitrarily formatted data info and turn > it > > into something that you can use in a template %$!mydate:::<format>% and > have > > it do the right thing. > > > >> Anyhow, I'll add a note to the github tracker so that we can see when > >> we go there. In general, I try to limit the core parsers, as in v2 we > >> now really have much better capabilities. > > > > > > I agree, but this is a case where you need to assemble multiple data > > elements that can be in different orders, and with different separators > (and > > sometimes month will be 3, 03, Mar, March) but you want the real result > to > > be internally a 'datetime' variable. > > OK, that means the short-term solution is actually to add an additional > parser. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
