is there a guide of sorts we can follow, or noodle through, to write our own java based parser?
or do we need to just java through and figure it out? ________________________________ From: Otto Fowler <[email protected]> Sent: Tuesday, October 17, 2017 1:30 PM To: Youzha; [email protected] Subject: Re: event correlation on metron So, There are several options parsing the data and enriching. 1. A native parser ( java ), which you have noticed is not there 2. An instance of the GROK parser, with GROK rules that parser the input 3. If it is CSV an instance of the CSV parser 4. If it is JSON an instance of the JSONMap parser If these cannot be applied to your file then your options are: 1. Write or open a jira for a native parser 2. find a way to transform your data to one of the above formats, so you can use those parsers. This again is where nifi can help. Something like: [nifi] TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA where TRANSFORM_TO_JSON is a script processor or something built in depending on your format. On October 17, 2017 at 13:16:05, Youzha ([email protected]<mailto:[email protected]>) wrote: Hi Lauren thx for your reply, yeah your suggestion absolutely right. i was able to ingest the logs to kafka. but how metron can enrich and index all of it? i think there are only bro, snort, yaf, snort, pcap, websphere topology storm on metron for parsers. so, how metron can read the logs telemetry and proccess it so i can use it to event correlation On Tue, 17 Oct 2017 at 23.11 Laurens Vets <[email protected]<mailto:[email protected]>> wrote: Hi Youzha, Either check how the snort logs on the full dev installation are ingested (I believe it's with a script) or check the Apache NiFi project which makes it very easy to read logs from almost any format and ingest them to Metron via Kafka. On 2017-10-17 08:53, Youzha wrote: is it possible to ingest other logs like /var/log/secure for example to be new telemetry on metron? i've seen the metron architecture on the website like picture below. host logs, email, av, etc can be telemetry event buffer on metron. if this possible, could you give me some suggestion how to do it ? [X] On Tue, 17 Oct 2017 at 21.00 Nick Allen <[email protected]<mailto:[email protected]>> wrote: If you want to look at failed login attempts for each user over time, then the Profiler might be a good solution. Your profile will depend on the fields available in your telemetry, but it would look something like this, as an example. { "profile": "failed-logins", "foreach": "user.name<https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fuser.name&data=02%7C01%7Cragdelaed%40hotmail.com%7Cfff6af2580d14dbe7cc108d51584d773%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636438582643979156&sdata=CAtDY7OiqSOr4217Br0AAS7oG2GBtd6kHU8wgxvFn4s%3D&reserved=0>", "onlyif": "source.type == 'activedirectory' and event.type == 'failed_login'" "init": { "count": 0 }, "update": { "count" : "count + 1" }, "result": "count" } You can find an introduction and more information on using the Profiler below. * https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fmetron%2Ftree%2Fmaster%2Fmetron-analytics%2Fmetron-profiler&data=02%7C01%7Cragdelaed%40hotmail.com%7Cfff6af2580d14dbe7cc108d51584d773%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636438582643979156&sdata=yqV3Vo%2FA766aAh5wfeCWX%2BH%2BhqQTMh4zveuIV%2BtJMJU%3D&reserved=0> * https://www.slideshare.net/secret/GFBf2RTXBG35PB<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.slideshare.net%2Fsecret%2FGFBf2RTXBG35PB&data=02%7C01%7Cragdelaed%40hotmail.com%7Cfff6af2580d14dbe7cc108d51584d773%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636438582643979156&sdata=9linbLpi2yvYFV4jtcA4sK5neMIggpGr4Lc2c%2BaGTwo%3D&reserved=0> Best of luck On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <[email protected]<mailto:[email protected]>> wrote: for example, i wanna try to correlate between logs. how many times user A have login failed and how many times user A have login succeed. include detail IP, timestamp etc. is this possible to do with metron? On 17/10/17 02:56, James Sirota wrote: What specifically are you looking to correlate? Can you talk a little more about your use case? 16.10.2017, 02:23, "tkg_cangkul" <[email protected]<mailto:[email protected]>>: hi, anyone could explain me about event correlation using apache metron? does metron support event correlation? Pls Advice ------------------- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
