Pieter, You can always create jira issues for things that you think are wrong or missing in the existing parsers, and maybe that work can get done.
There are also things ‘in the pipeline’ that you may want to think about. - There is a new regex parser that just landed. - There is a syslog 3164 parser that is in pr, that includes refactoring of the 5424 parser such that these parsers can be derived from and used by the ’syslog’ based parsers, that parse the syslog themselves ( making them simpler ). CEF is a candidate to be redone on this base, and if you do your own, you may want to think about that approach too. On December 18, 2018 at 01:59:13, Pieter Baele ([email protected]) wrote: Hi, Thank you for the welcome! The concepts behind Metron are nice, very nice. Finally network log / data analysis is possible using different approaches. It has been a few years since I used tools such as Suricata, Bro, Snort, OSSEC, PF_RING. But the integration of some of those at scale... nice possibilities! In our environment we only have some krb issues, maybe related to our specific choices, but I am sure we will be able to solve those.... I think our use cases can perfectly work with the defaults. We probably need to get the necessary experience. Both in parsing, using Stellar, writing Elasticsearch templates, Spark.... So basically If I set the timestamp field correctly, we can correlate and profile with other events around the same timestamp, right? We could also write a 3th party parser , almost a copy of the CEF parser and modify it specifically for this device (and contribute back...)? But if it is only about the timestamp, probably only the fieldTransformation is necessary. I suppose it also by design that the ASA parser only does some 'basic' parsing? Currently there is no interpretation of access-list, hit counts, etcetera. But I thought it was better using a Java parser than grok patterns if possible, because of optimalisations....? Pieter On Mon, Dec 17, 2018 at 5:49 PM Simon Elliston Ball < [email protected]> wrote: > Hi Pieter, > > Welcome to the Metron community. > > The logic used by the CEF parser right now is to populate the timestamp > field as follows: > 1. if the rt field exists, use that. > 2. if there is a field matching the syslogTime pattern (either the old > pattern of 5424) > 3. If neither are present, it uses current system time (as a last resort) > > You can use stellar fieldTransformations after the fact to change the > timestamp field to map to anything you like, including any parsed field. > > The field type is ignored in custom fields from CEF, since type is not a > required construct in the Metron JSON object, so we just take the > corresponding label field and map the value to that, since the Metron > schema is more flexible than the CEF schema. I can see there may be edge > cases where this could create a collision, and would be interested to hear > about your use case around this, and any problems that approach might > cause, but it seems to have worked for most people so far. > > Simon > > > On Mon, 17 Dec 2018 at 15:38, Pieter Baele <[email protected]> wrote: > >> Hi, >> >> For correlation & profiling the presense of a correct timestamp / >> eventtime is important, >> what to do with a device implementing CEF output, but not properly >> providing the rt field? >> Also syslogTime is not parsed by the CEF parser. >> >> There is another field present, how can I assure this field is taken as >> timestamp for further analysis and ingest in ES and HDFS? >> >> Also, is it not necessary to have the type of a field set during (CEF) >> parsing (maybe I am missing something there) >> >> Sincerely >> Pieter >> >> >> >> > > -- > -- > simon elliston ball > @sireb >
