What you want to do in this setting is just TailFile, the just push to Kafka. The grok piece is more efficiently handled in the Metron grok parser.
Push to a kafka topic named for your sensor, then setup a sensor (a parser topology to do the grok parsing and any transformation you need). Each sensor gets its own parser topology. Simon > On 17 Oct 2017, at 19:00, Youzha <[email protected]> wrote: > > after nifi procces : > > TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA > > what metron topology that i can use to procces the data in kafka? so it can > be enrichment by metron. i’ve check the article about adding new telemetry > source with squid, there is a squid topology that will ingest from the squid > topic in kafka and then put on enrichment kafka topic. > so how about my use case above? is there any topology that i can use? > >> On Wed, 18 Oct 2017 at 00.30 Otto Fowler <[email protected]> wrote: >> So, >> There are several options parsing the data and enriching. >> >> 1. A native parser ( java ), which you have noticed is not there >> 2. An instance of the GROK parser, with GROK rules that parser the input >> 3. If it is CSV an instance of the CSV parser >> 4. If it is JSON an instance of the JSONMap parser >> >> If these cannot be applied to your file then your options are: >> >> 1. Write or open a jira for a native parser >> 2. find a way to transform your data to one of the above formats, so you can >> use those parsers. This again is where nifi can help. Something like: >> >> >> [nifi] >> >> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA >> >> where TRANSFORM_TO_JSON is a script processor or something built in >> depending on your format. >> >> >> >>> On October 17, 2017 at 13:16:05, Youzha ([email protected]) wrote: >>> >>> Hi Lauren thx for your reply, >>> >>> yeah your suggestion absolutely right. i was able to ingest the logs to >>> kafka. but how metron can enrich and index all of it? i think there are >>> only bro, snort, yaf, snort, pcap, websphere topology storm on metron for >>> parsers. so, how metron can read the logs telemetry and proccess it so i >>> can use it to event correlation >>> >>>> On Tue, 17 Oct 2017 at 23.11 Laurens Vets <[email protected]> wrote: >>>> Hi Youzha, >>>> >>>> Either check how the snort logs on the full dev installation are ingested >>>> (I believe it's with a script) or check the Apache NiFi project which >>>> makes it very easy to read logs from almost any format and ingest them to >>>> Metron via Kafka. >>>> >>>>> On 2017-10-17 08:53, Youzha wrote: >>>>> >>>>> is it possible to ingest other logs like /var/log/secure for example to >>>>> be new telemetry on metron? i've seen the metron architecture on the >>>>> website like picture below. host logs, email, av, etc can be telemetry >>>>> event buffer on metron. if this possible, could you give me some >>>>> suggestion how to do it ? >>>>> >>>>> >>>>>> On Tue, 17 Oct 2017 at 21.00 Nick Allen <[email protected]> wrote: >>>>>> If you want to look at failed login attempts for each user over time, >>>>>> then the Profiler might be a good solution. Your profile will depend on >>>>>> the fields available in your telemetry, but it would look something like >>>>>> this, as an example. >>>>>> >>>>>> { >>>>>> "profile": "failed-logins", >>>>>> "foreach": "user.name", >>>>>> "onlyif": "source.type == 'activedirectory' and event.type == >>>>>> 'failed_login'" >>>>>> "init": { "count": 0 }, >>>>>> "update": { "count" : "count + 1" }, >>>>>> "result": "count" >>>>>> } >>>>>> >>>>>> You can find an introduction and more information on using the Profiler >>>>>> below. >>>>>> * >>>>>> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler >>>>>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB >>>>>> >>>>>> Best of luck >>>>>> >>>>>>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <[email protected]> >>>>>>> wrote: >>>>>>> for example, >>>>>>> >>>>>>> i wanna try to correlate between logs. >>>>>>> how many times user A have login failed and how many times user A have >>>>>>> login succeed. include detail IP, timestamp etc. >>>>>>> is this possible to do with metron? >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On 17/10/17 02:56, James Sirota wrote: >>>>>>>> What specifically are you looking to correlate? Can you talk a little >>>>>>>> more about your use case? >>>>>>>> >>>>>>>> 16.10.2017, 02:23, "tkg_cangkul" <[email protected]>: >>>>>>>>> hi, >>>>>>>>> >>>>>>>>> anyone could explain me about event correlation using apache metron? >>>>>>>>> does metron support event correlation? >>>>>>>>> >>>>>>>>> Pls Advice >>>>>>>> ------------------- >>>>>>>> Thank you, >>>>>>>> >>>>>>>> James Sirota >>>>>>>> PMC- Apache Metron >>>>>>>> jsirota AT apache DOT org >>>>
