On 23. Juni 2016 at 19:48:30, [email protected] ([email protected]) wrote: > I am trying to get graylog to interrupt a field I am sending over a field > that I would like to interpret as a timestamp however the issue that I am > having is that it is coming across as UNIX epoch > > 1) Is there an operation I can do on the graylog to convert this as a > datetimestamp > 2) Once converted is there a way to have this data searchable as a > datetimestamp just like the current timestamp variable
I just copy over the Conversation we had in IRC about this: [19:40:18] <ghanima> hello all [19:41:24] <ghanima> I am trying to get graylog to interrupt a field I am sending over a field that I would like to interpret as a timestamp however the issue that I am having is that it is coming across as UNIX epoch [19:41:49] <ghanima> 1) Is there an operation I can do on the graylog to convert this as a datetimestamp [19:42:18] <ghanima> 2) Once converted is there a way to have this data searchable as a datetimestamp just like the current timestamp variable [19:52:08] <jalogisch> ghanima: yes and yes [19:52:49] <ghanima> jalogisch: I am all ears on how to approach this but I am not sure where to start [19:53:14] <ghanima> How can I convert the epoch to a datetime within graylog [19:53:40] <jalogisch> how does a log that contains this look like? [19:54:30] <ghanima> jalogisch: the entries are being pulled from a file [19:54:34] <ghanima> sample entry looks like this [19:54:35] <ghanima> 16/06/20 22:30:56 WARN InfluxDBQuarantineHandler: Message quarantined! Reason: Invalid - reading time ahead of current clock time, Msg: 47314fd5-5468-4d9e-b051-30015a474916.fb55df42de304c6a57421da3218a7c54-CAT.03f72667-15dd-4587-a180-c248a06bde4e.02ed558e-73ab-4807-876c-d0b69b255645 1466490475000 171984.0 [19:54:43] <jalogisch> first option, extract via grok http://docs.graylog.org/en/2.0/pages/extractors.html?highlight=grok#using-grok-patterns-to-extract-data and create a new field that gets converted into a timestamp [19:55:22] <jalogisch> in the format you like to have [19:55:40] <ghanima> I have configured on the graylog side a grok filter %{DATESTAMP:insertdate} %{WORD:logstatus} %{WORD:influx}: (?<influxmessage>[a-zA-Z]*\s[a-zA-Z]*)! (?<platformerror>.*), Msg: (?<orgID>\w*-\w*-\w*-\w*-\w*).(?<AssetID>\w*-CAT).(?<ComponentStructureID>\w*-\w*-\w*-\w*-\w*).(?<metricID>\w*-\w*-\w*-\w*-\w*) %{NUMBER:metrictimestamp} (?<metricvalue>.*) [19:56:26] <ghanima> I want this field 1466490475000 which is the 3rd to last be converted to MM/DD/YY hh:mm:ss Z [19:58:09] <ghanima> jalogisch: does that make sense [19:58:50] <jalogisch> second option is to use the http://docs.graylog.org/en/2.0/pages/extractors.html?highlight=grok#the-standard-date-converter date converter after extraction [20:00:01] <jalogisch> looks valid [20:03:02] <jalogisch> replace %{NUMBER:..} with %{NUMBER:metrictimestamp:timestamp;date;dd/MMM/yyyy:HH:mm:ss Z} [20:03:31] <jalogisch> that should do the trick - as written a few lines above http://docs.graylog.org/en/2.0/pages/extractors.html?highlight=grok#using-the-json-extractor [20:03:53] <jalogisch> means you need to scroll up a few lines to have this information [20:04:55] <ghanima> jalogisch: what about all the data that has been index is there a way to re-process that data [20:05:21] <jalogisch> not within graylog [20:05:50] <jalogisch> you would need to modify direct in elasticsearch or export the data and send it again to graylog [20:07:26] <ghanima> jalogisch: in my grok pattern %{DATESTAMP:insertdate} [20:07:52] <ghanima> can I trust that this will store this data as a date timestamp and its searchable as such or is there another conversion that needs to be done [20:09:14] <jalogisch> read the docs - it is explained. for date you can grep, store in a new field and convert with one grok [20:09:33] <jalogisch> but you need to specific the format you like to have as a result [20:15:37] <ghanima> jalogisch: so unless I misread you posted this is what happens [20:15:41] <ghanima> when I apply this grok %{NUMBER:metrictimestamp;date;MMMM/dd/yyyy:HH:mm:ss Z} [20:15:46] <ghanima> I get this error [20:16:05] <ghanima> java.text.ParseException: Unparseable date: "1466210640000" [20:17:01] <ghanima> jalogisch: I tried both NUMBER and DATA [20:22:25] <jalogisch> i see - and checked the configuration [20:22:45] <jalogisch> SimpleDateFormat is the range for this conversion [20:23:01] <jalogisch> and that does not see epoch as a valid date format [20:23:37] <jalogisch> can you please fill an issue https://github.com/Graylog2/graylog2-server/issues that this get corrected [20:26:11] <jalogisch> and to solve your issue you will need to try the Flexibly parse date extractor with a copy input extractor that contains the data -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAGm-bLYZ-BzNqRNDG5x%2B%3DKRCKrXvT2k3d6ak_Kc5YBLaDJfk%3Dg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
