Re: event correlation on metron

[Youzha]

is there any documentation to create new sensor in metron?

On Wed, 18 Oct 2017 at 01.22 Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Best bet there is to create a new sensor config using the grok parser
> type. So you would for example have a kafka topic called host_dhcp and a
> sensor called host_dhcp with the relevant grok pattern.
>
> Simon
>
>
> On 17 Oct 2017, at 19:19, Youzha  wrote:
>
> that’s what i mean.
> what sensor that i need if i want to do this case?
> especially when i wanna parse some host logs into metron enrichment and
> indexing
>
> On Wed, 18 Oct 2017 at 01.03 Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>> What you want to do in this setting is just TailFile, the just push to
>> Kafka. The grok piece is more efficiently handled in the Metron grok parser.
>>
>> Push to a kafka topic named for your sensor, then setup a sensor (a
>> parser topology to do the grok parsing and any transformation you need).
>> Each sensor gets its own parser topology.
>>
>> Simon
>>
>>
>> On 17 Oct 2017, at 19:00, Youzha  wrote:
>>
>> after nifi procces :
>>
>> TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
>>
>> what metron topology that i can use to procces the data in kafka? so it
>> can be enrichment by metron. i’ve check the article about adding new
>> telemetry source with squid, there is a squid topology that will ingest
>> from the squid topic in kafka and then put on enrichment kafka topic.
>> so how about my use case above? is there any topology that i can use?
>>
>> On Wed, 18 Oct 2017 at 00.30 Otto Fowler  wrote:
>>
>>> So,
>>> There are several options parsing the data and enriching.
>>>
>>> 1.  A native parser ( java ), which you have noticed is not there
>>> 2.  An instance of the GROK parser, with GROK rules that parser the input
>>> 3.  If it is CSV an instance of the CSV parser
>>> 4.  If it is JSON an instance of the JSONMap parser
>>>
>>> If these cannot be applied to your file then your options are:
>>>
>>> 1.  Write or open a jira for a native parser
>>> 2. find a way to transform your data to one of the above formats, so you
>>> can use those parsers.  This again is where nifi can help.  Something like:
>>>
>>>
>>> [nifi]
>>>
>>> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
>>>
>>> where TRANSFORM_TO_JSON is a script processor or something built in
>>> depending on your format.
>>>
>>>
>>>
>>> On October 17, 2017 at 13:16:05, Youzha (yuza.ras...@gmail.com) wrote:
>>>
>>> Hi Lauren thx for your reply,
>>>
>>> yeah your suggestion absolutely right. i was able to ingest the logs to
>>> kafka. but how metron can enrich and index all of it? i think there are
>>> only  bro, snort, yaf, snort, pcap, websphere topology storm on metron for
>>> parsers. so, how metron can read the logs telemetry and proccess it so i
>>> can use it to event correlation
>>>
>>> On Tue, 17 Oct 2017 at 23.11 Laurens Vets  wrote:
>>>
 Hi Youzha,

 Either check how the snort logs on the full dev installation are
 ingested (I believe it's with a script) or check the Apache NiFi project
 which makes it very easy to read logs from almost any format and ingest
 them to Metron via Kafka.

 On 2017-10-17 08:53, Youzha wrote:

 is it possible to ingest other logs like /var/log/secure for example to
 be new telemetry on metron? i've seen the metron architecture on the
 website like picture below. host logs, email, av, etc can be telemetry
 event buffer on metron. if this possible, could you give me some suggestion
 how to do it ?


 On Tue, 17 Oct 2017 at 21.00 Nick Allen  wrote:

> If you want to look at failed login attempts for each user over time,
> then the Profiler might be a good solution.  Your profile will depend on
> the fields available in your telemetry, but it would look something like
> this, as an example.
>
>
> {
>   "profile": "failed-logins",
>   "foreach": "user.name",
>   "onlyif": "source.type == 'activedirectory' and event.type ==
> 'failed_login'"
>   "init": { "count": 0 },
>   "update": { "count" : "count + 1" },
>   "result": "count"
> }
>
>
> You can find an introduction and more information on using the
> Profiler below.
> *
> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>
> Best of luck
>
> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul 
> wrote:
>
>> for example,
>>
>> i wanna try to correlate between logs.
>> how many times user A have login failed and how many times user A
>> have login succeed. include detail IP, timestamp etc.
>> is this possible to do with metron?
>>
>>
>>
>>
>> On 17/10/17 02:56, James Sirota wrote:
>>

Re: event correlation on metron

[Simon Elliston Ball]

Best bet there is to create a new sensor config using the grok parser type. So 
you would for example have a kafka topic called host_dhcp and a sensor called 
host_dhcp with the relevant grok pattern. 

Simon 

> On 17 Oct 2017, at 19:19, Youzha  wrote:
> 
> that’s what i mean.
> what sensor that i need if i want to do this case? 
> especially when i wanna parse some host logs into metron enrichment and 
> indexing
> 
>> On Wed, 18 Oct 2017 at 01.03 Simon Elliston Ball 
>>  wrote:
>> What you want to do in this setting is just TailFile, the just push to 
>> Kafka. The grok piece is more efficiently handled in the Metron grok parser.
>> 
>> Push to a kafka topic named for your sensor, then setup a sensor (a parser 
>> topology to do the grok parsing and any transformation you need). Each 
>> sensor gets its own parser topology.
>> 
>> Simon 
>> 
>> 
>>> On 17 Oct 2017, at 19:00, Youzha  wrote:
>>> 
>>> after nifi procces :
>>> 
>>> TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
>>> 
>>> what metron topology that i can use to procces the data in kafka? so it can 
>>> be enrichment by metron. i’ve check the article about adding new telemetry 
>>> source with squid, there is a squid topology that will ingest from the 
>>> squid topic in kafka and then put on enrichment kafka topic. 
>>> so how about my use case above? is there any topology that i can use?
>>> 
 On Wed, 18 Oct 2017 at 00.30 Otto Fowler  wrote:
 So, 
 There are several options parsing the data and enriching.
 
 1.  A native parser ( java ), which you have noticed is not there
 2.  An instance of the GROK parser, with GROK rules that parser the input
 3.  If it is CSV an instance of the CSV parser
 4.  If it is JSON an instance of the JSONMap parser
 
 If these cannot be applied to your file then your options are:
 
 1.  Write or open a jira for a native parser
 2. find a way to transform your data to one of the above formats, so you 
 can use those parsers.  This again is where nifi can help.  Something like:
 
 
 [nifi]
 
 TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
 
 where TRANSFORM_TO_JSON is a script processor or something built in 
 depending on your format.
 
 
 
> On October 17, 2017 at 13:16:05, Youzha (yuza.ras...@gmail.com) wrote:
> 
> Hi Lauren thx for your reply,
> 
> yeah your suggestion absolutely right. i was able to ingest the logs to 
> kafka. but how metron can enrich and index all of it? i think there are 
> only  bro, snort, yaf, snort, pcap, websphere topology storm on metron 
> for parsers. so, how metron can read the logs telemetry and proccess it 
> so i can use it to event correlation
> 
>> On Tue, 17 Oct 2017 at 23.11 Laurens Vets  wrote:
>> Hi Youzha,
>> 
>> Either check how the snort logs on the full dev installation are 
>> ingested (I believe it's with a script) or check the Apache NiFi project 
>> which makes it very easy to read logs from almost any format and ingest 
>> them to Metron via Kafka.
>> 
>>> On 2017-10-17 08:53, Youzha wrote:
>>> 
>>> is it possible to ingest other logs like /var/log/secure for example to 
>>> be new telemetry on metron? i've seen the metron architecture on the 
>>> website like picture below. host logs, email, av, etc can be telemetry 
>>> event buffer on metron. if this possible, could you give me some 
>>> suggestion how to do it ?
>>>  
>>> 
 On Tue, 17 Oct 2017 at 21.00 Nick Allen  wrote:
 If you want to look at failed login attempts for each user over time, 
 then the Profiler might be a good solution.  Your profile will depend 
 on the fields available in your telemetry, but it would look something 
 like this, as an example.
  
 {
   "profile": "failed-logins",
   "foreach": "user.name",
   "onlyif": "source.type == 'activedirectory' and event.type == 
 'failed_login'"
   "init": { "count": 0 },
   "update": { "count" : "count + 1" },
   "result": "count"
 }
  
 You can find an introduction and more information on using the 
 Profiler below.
 * 
 https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
 * https://www.slideshare.net/secret/GFBf2RTXBG35PB
  
 Best of luck
 
> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul  
> wrote:
> for example,
> 
> i wanna try to correlate between logs.
> how many times user A have login failed and how many times user A 
> have login succeed. include detail IP, timestamp etc.
> is this possible to do with 

Re: event correlation on metron

[Youzha]

that’s what i mean.
what sensor that i need if i want to do this case?
especially when i wanna parse some host logs into metron enrichment and
indexing

On Wed, 18 Oct 2017 at 01.03 Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> What you want to do in this setting is just TailFile, the just push to
> Kafka. The grok piece is more efficiently handled in the Metron grok parser.
>
> Push to a kafka topic named for your sensor, then setup a sensor (a parser
> topology to do the grok parsing and any transformation you need). Each
> sensor gets its own parser topology.
>
> Simon
>
>
> On 17 Oct 2017, at 19:00, Youzha  wrote:
>
> after nifi procces :
>
> TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
>
> what metron topology that i can use to procces the data in kafka? so it
> can be enrichment by metron. i’ve check the article about adding new
> telemetry source with squid, there is a squid topology that will ingest
> from the squid topic in kafka and then put on enrichment kafka topic.
> so how about my use case above? is there any topology that i can use?
>
> On Wed, 18 Oct 2017 at 00.30 Otto Fowler  wrote:
>
>> So,
>> There are several options parsing the data and enriching.
>>
>> 1.  A native parser ( java ), which you have noticed is not there
>> 2.  An instance of the GROK parser, with GROK rules that parser the input
>> 3.  If it is CSV an instance of the CSV parser
>> 4.  If it is JSON an instance of the JSONMap parser
>>
>> If these cannot be applied to your file then your options are:
>>
>> 1.  Write or open a jira for a native parser
>> 2. find a way to transform your data to one of the above formats, so you
>> can use those parsers.  This again is where nifi can help.  Something like:
>>
>>
>> [nifi]
>>
>> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
>>
>> where TRANSFORM_TO_JSON is a script processor or something built in
>> depending on your format.
>>
>>
>>
>> On October 17, 2017 at 13:16:05, Youzha (yuza.ras...@gmail.com) wrote:
>>
>> Hi Lauren thx for your reply,
>>
>> yeah your suggestion absolutely right. i was able to ingest the logs to
>> kafka. but how metron can enrich and index all of it? i think there are
>> only  bro, snort, yaf, snort, pcap, websphere topology storm on metron for
>> parsers. so, how metron can read the logs telemetry and proccess it so i
>> can use it to event correlation
>>
>> On Tue, 17 Oct 2017 at 23.11 Laurens Vets  wrote:
>>
>>> Hi Youzha,
>>>
>>> Either check how the snort logs on the full dev installation are
>>> ingested (I believe it's with a script) or check the Apache NiFi project
>>> which makes it very easy to read logs from almost any format and ingest
>>> them to Metron via Kafka.
>>>
>>> On 2017-10-17 08:53, Youzha wrote:
>>>
>>> is it possible to ingest other logs like /var/log/secure for example to
>>> be new telemetry on metron? i've seen the metron architecture on the
>>> website like picture below. host logs, email, av, etc can be telemetry
>>> event buffer on metron. if this possible, could you give me some suggestion
>>> how to do it ?
>>>
>>>
>>> On Tue, 17 Oct 2017 at 21.00 Nick Allen  wrote:
>>>
 If you want to look at failed login attempts for each user over time,
 then the Profiler might be a good solution.  Your profile will depend on
 the fields available in your telemetry, but it would look something like
 this, as an example.


 {
   "profile": "failed-logins",
   "foreach": "user.name",
   "onlyif": "source.type == 'activedirectory' and event.type ==
 'failed_login'"
   "init": { "count": 0 },
   "update": { "count" : "count + 1" },
   "result": "count"
 }


 You can find an introduction and more information on using the Profiler
 below.
 *
 https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
 * https://www.slideshare.net/secret/GFBf2RTXBG35PB

 Best of luck

 On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul 
 wrote:

> for example,
>
> i wanna try to correlate between logs.
> how many times user A have login failed and how many times user A have
> login succeed. include detail IP, timestamp etc.
> is this possible to do with metron?
>
>
>
>
> On 17/10/17 02:56, James Sirota wrote:
>
>> What specifically are you looking to correlate?  Can you talk a
>> little more about your use case?
>>
>> 16.10.2017, 02:23, "tkg_cangkul" :
>>
>>> hi,
>>>
>>> anyone could explain me about event correlation using apache metron?
>>> does metron support event correlation?
>>>
>>> Pls Advice
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>
>
>>>

Re: event correlation on metron

[Simon Elliston Ball]

What you want to do in this setting is just TailFile, the just push to Kafka. 
The grok piece is more efficiently handled in the Metron grok parser.

Push to a kafka topic named for your sensor, then setup a sensor (a parser 
topology to do the grok parsing and any transformation you need). Each sensor 
gets its own parser topology.

Simon 

> On 17 Oct 2017, at 19:00, Youzha  wrote:
> 
> after nifi procces :
> 
> TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
> 
> what metron topology that i can use to procces the data in kafka? so it can 
> be enrichment by metron. i’ve check the article about adding new telemetry 
> source with squid, there is a squid topology that will ingest from the squid 
> topic in kafka and then put on enrichment kafka topic. 
> so how about my use case above? is there any topology that i can use?
> 
>> On Wed, 18 Oct 2017 at 00.30 Otto Fowler  wrote:
>> So, 
>> There are several options parsing the data and enriching.
>> 
>> 1.  A native parser ( java ), which you have noticed is not there
>> 2.  An instance of the GROK parser, with GROK rules that parser the input
>> 3.  If it is CSV an instance of the CSV parser
>> 4.  If it is JSON an instance of the JSONMap parser
>> 
>> If these cannot be applied to your file then your options are:
>> 
>> 1.  Write or open a jira for a native parser
>> 2. find a way to transform your data to one of the above formats, so you can 
>> use those parsers.  This again is where nifi can help.  Something like:
>> 
>> 
>> [nifi]
>> 
>> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
>> 
>> where TRANSFORM_TO_JSON is a script processor or something built in 
>> depending on your format.
>> 
>> 
>> 
>>> On October 17, 2017 at 13:16:05, Youzha (yuza.ras...@gmail.com) wrote:
>>> 
>>> Hi Lauren thx for your reply,
>>> 
>>> yeah your suggestion absolutely right. i was able to ingest the logs to 
>>> kafka. but how metron can enrich and index all of it? i think there are 
>>> only  bro, snort, yaf, snort, pcap, websphere topology storm on metron for 
>>> parsers. so, how metron can read the logs telemetry and proccess it so i 
>>> can use it to event correlation
>>> 
 On Tue, 17 Oct 2017 at 23.11 Laurens Vets  wrote:
 Hi Youzha,
 
 Either check how the snort logs on the full dev installation are ingested 
 (I believe it's with a script) or check the Apache NiFi project which 
 makes it very easy to read logs from almost any format and ingest them to 
 Metron via Kafka.
 
> On 2017-10-17 08:53, Youzha wrote:
> 
> is it possible to ingest other logs like /var/log/secure for example to 
> be new telemetry on metron? i've seen the metron architecture on the 
> website like picture below. host logs, email, av, etc can be telemetry 
> event buffer on metron. if this possible, could you give me some 
> suggestion how to do it ?
>  
> 
>> On Tue, 17 Oct 2017 at 21.00 Nick Allen  wrote:
>> If you want to look at failed login attempts for each user over time, 
>> then the Profiler might be a good solution.  Your profile will depend on 
>> the fields available in your telemetry, but it would look something like 
>> this, as an example.
>>  
>> {
>>   "profile": "failed-logins",
>>   "foreach": "user.name",
>>   "onlyif": "source.type == 'activedirectory' and event.type == 
>> 'failed_login'"
>>   "init": { "count": 0 },
>>   "update": { "count" : "count + 1" },
>>   "result": "count"
>> }
>>  
>> You can find an introduction and more information on using the Profiler 
>> below.
>> * 
>> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>>  
>> Best of luck
>> 
>>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul  
>>> wrote:
>>> for example,
>>> 
>>> i wanna try to correlate between logs.
>>> how many times user A have login failed and how many times user A have 
>>> login succeed. include detail IP, timestamp etc.
>>> is this possible to do with metron?
>>> 
>>> 
>>> 
>>> 
 On 17/10/17 02:56, James Sirota wrote:
 What specifically are you looking to correlate?  Can you talk a little 
 more about your use case?
 
 16.10.2017, 02:23, "tkg_cangkul" :
> hi,
> 
> anyone could explain me about event correlation using apache metron?
> does metron support event correlation?
> 
> Pls Advice
 ---
 Thank you,
 
 James Sirota
 PMC- Apache Metron
 jsirota AT apache DOT org
 

Re: event correlation on metron

[ed d]

is there a guide of sorts we can follow, or noodle through, to write our own 
java based parser?


or do we need to just java through and figure it out?



From: Otto Fowler 
Sent: Tuesday, October 17, 2017 1:30 PM
To: Youzha; user@metron.apache.org
Subject: Re: event correlation on metron

So,
There are several options parsing the data and enriching.

1.  A native parser ( java ), which you have noticed is not there
2.  An instance of the GROK parser, with GROK rules that parser the input
3.  If it is CSV an instance of the CSV parser
4.  If it is JSON an instance of the JSONMap parser

If these cannot be applied to your file then your options are:

1.  Write or open a jira for a native parser
2. find a way to transform your data to one of the above formats, so you can 
use those parsers.  This again is where nifi can help.  Something like:


[nifi]

TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA

where TRANSFORM_TO_JSON is a script processor or something built in depending 
on your format.




On October 17, 2017 at 13:16:05, Youzha 
(yuza.ras...@gmail.com) wrote:

Hi Lauren thx for your reply,

yeah your suggestion absolutely right. i was able to ingest the logs to kafka. 
but how metron can enrich and index all of it? i think there are only  bro, 
snort, yaf, snort, pcap, websphere topology storm on metron for parsers. so, 
how metron can read the logs telemetry and proccess it so i can use it to event 
correlation

On Tue, 17 Oct 2017 at 23.11 Laurens Vets 
> wrote:

Hi Youzha,

Either check how the snort logs on the full dev installation are ingested (I 
believe it's with a script) or check the Apache NiFi project which makes it 
very easy to read logs from almost any format and ingest them to Metron via 
Kafka.

On 2017-10-17 08:53, Youzha wrote:

is it possible to ingest other logs like /var/log/secure for example to be new 
telemetry on metron? i've seen the metron architecture on the website like 
picture below. host logs, email, av, etc can be telemetry event buffer on 
metron. if this possible, could you give me some suggestion how to do it ?

[X]
On Tue, 17 Oct 2017 at 21.00 Nick Allen 
> wrote:
If you want to look at failed login attempts for each user over time, then the 
Profiler might be a good solution.  Your profile will depend on the fields 
available in your telemetry, but it would look something like this, as an 
example.

{
  "profile": "failed-logins",
  "foreach": 
"user.name",
  "onlyif": "source.type == 'activedirectory' and event.type == 'failed_login'"
  "init": { "count": 0 },
  "update": { "count" : "count + 1" },
  "result": "count"
}

You can find an introduction and more information on using the Profiler below.
* 
https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
* 
https://www.slideshare.net/secret/GFBf2RTXBG35PB

Best of luck

On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul 
> wrote:
for example,

i wanna try to correlate between logs.
how many times user A have login failed and how many times user A have login 
succeed. include detail IP, timestamp etc.
is this possible to do with metron?




On 17/10/17 02:56, James Sirota wrote:
What specifically are you looking to correlate?  Can you talk a little more 
about your use case?

16.10.2017, 02:23, "tkg_cangkul" 
>:
hi,

anyone could explain me about event correlation using apache metron?
does metron support event correlation?

Pls Advice
---
Thank you,

James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

Re: event correlation on metron

[Otto Fowler]

So,
There are several options parsing the data and enriching.

1.  A native parser ( java ), which you have noticed is not there
2.  An instance of the GROK parser, with GROK rules that parser the input
3.  If it is CSV an instance of the CSV parser
4.  If it is JSON an instance of the JSONMap parser

If these cannot be applied to your file then your options are:

1.  Write or open a jira for a native parser
2. find a way to transform your data to one of the above formats, so you
can use those parsers.  This again is where nifi can help.  Something like:


[nifi]

TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA

where TRANSFORM_TO_JSON is a script processor or something built in
depending on your format.



On October 17, 2017 at 13:16:05, Youzha (yuza.ras...@gmail.com) wrote:

Hi Lauren thx for your reply,

yeah your suggestion absolutely right. i was able to ingest the logs to
kafka. but how metron can enrich and index all of it? i think there are
only  bro, snort, yaf, snort, pcap, websphere topology storm on metron for
parsers. so, how metron can read the logs telemetry and proccess it so i
can use it to event correlation

On Tue, 17 Oct 2017 at 23.11 Laurens Vets  wrote:

> Hi Youzha,
>
> Either check how the snort logs on the full dev installation are ingested
> (I believe it's with a script) or check the Apache NiFi project which makes
> it very easy to read logs from almost any format and ingest them to Metron
> via Kafka.
>
> On 2017-10-17 08:53, Youzha wrote:
>
> is it possible to ingest other logs like /var/log/secure for example to be
> new telemetry on metron? i've seen the metron architecture on the website
> like picture below. host logs, email, av, etc can be telemetry event buffer
> on metron. if this possible, could you give me some suggestion how to do it
> ?
>
>
> On Tue, 17 Oct 2017 at 21.00 Nick Allen  wrote:
>
>> If you want to look at failed login attempts for each user over time,
>> then the Profiler might be a good solution.  Your profile will depend on
>> the fields available in your telemetry, but it would look something like
>> this, as an example.
>>
>>
>> {
>>   "profile": "failed-logins",
>>   "foreach": "user.name",
>>   "onlyif": "source.type == 'activedirectory' and event.type ==
>> 'failed_login'"
>>   "init": { "count": 0 },
>>   "update": { "count" : "count + 1" },
>>   "result": "count"
>> }
>>
>>
>> You can find an introduction and more information on using the Profiler
>> below.
>> *
>> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>>
>> Best of luck
>>
>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul 
>> wrote:
>>
>>> for example,
>>>
>>> i wanna try to correlate between logs.
>>> how many times user A have login failed and how many times user A have
>>> login succeed. include detail IP, timestamp etc.
>>> is this possible to do with metron?
>>>
>>>
>>>
>>>
>>> On 17/10/17 02:56, James Sirota wrote:
>>>
 What specifically are you looking to correlate?  Can you talk a little
 more about your use case?

 16.10.2017, 02:23, "tkg_cangkul" :

> hi,
>
> anyone could explain me about event correlation using apache metron?
> does metron support event correlation?
>
> Pls Advice

 ---
 Thank you,

 James Sirota
 PMC- Apache Metron
 jsirota AT apache DOT org
>>>
>>>
>

Re: event correlation on metron

[Youzha]

Hi Lauren thx for your reply,

yeah your suggestion absolutely right. i was able to ingest the logs to
kafka. but how metron can enrich and index all of it? i think there are
only  bro, snort, yaf, snort, pcap, websphere topology storm on metron for
parsers. so, how metron can read the logs telemetry and proccess it so i
can use it to event correlation

On Tue, 17 Oct 2017 at 23.11 Laurens Vets  wrote:

> Hi Youzha,
>
> Either check how the snort logs on the full dev installation are ingested
> (I believe it's with a script) or check the Apache NiFi project which makes
> it very easy to read logs from almost any format and ingest them to Metron
> via Kafka.
>
> On 2017-10-17 08:53, Youzha wrote:
>
> is it possible to ingest other logs like /var/log/secure for example to be
> new telemetry on metron? i've seen the metron architecture on the website
> like picture below. host logs, email, av, etc can be telemetry event buffer
> on metron. if this possible, could you give me some suggestion how to do it
> ?
>
>
> On Tue, 17 Oct 2017 at 21.00 Nick Allen  wrote:
>
>> If you want to look at failed login attempts for each user over time,
>> then the Profiler might be a good solution.  Your profile will depend on
>> the fields available in your telemetry, but it would look something like
>> this, as an example.
>>
>>
>> {
>>   "profile": "failed-logins",
>>   "foreach": "user.name",
>>   "onlyif": "source.type == 'activedirectory' and event.type ==
>> 'failed_login'"
>>   "init": { "count": 0 },
>>   "update": { "count" : "count + 1" },
>>   "result": "count"
>> }
>>
>>
>> You can find an introduction and more information on using the Profiler
>> below.
>> *
>> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>>
>> Best of luck
>>
>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul 
>> wrote:
>>
>>> for example,
>>>
>>> i wanna try to correlate between logs.
>>> how many times user A have login failed and how many times user A have
>>> login succeed. include detail IP, timestamp etc.
>>> is this possible to do with metron?
>>>
>>>
>>>
>>>
>>> On 17/10/17 02:56, James Sirota wrote:
>>>
 What specifically are you looking to correlate?  Can you talk a little
 more about your use case?

 16.10.2017, 02:23, "tkg_cangkul" :

> hi,
>
> anyone could explain me about event correlation using apache metron?
> does metron support event correlation?
>
> Pls Advice

 ---
 Thank you,

 James Sirota
 PMC- Apache Metron
 jsirota AT apache DOT org
>>>
>>>
>

Re: Snort

[Syed Hammad Tahir]

I am so noob in all of this. I am using full-dev vm metron install to do my
research. So I have 2 options to install snort: as per my understanding

1- Install it in a usual way (like that on a regular linux machine) and
then make its kafka topic

2- Use ansible role to do all of that. Read the content of those yml files
given in main.yml to understand the procedure?

Which one do you suggest?



On Tue, Oct 17, 2017 at 9:22 PM, Nick Allen  wrote:

> No special commands.  Install and configure Snort however you like and get
> those logs into a Kafka topic.  Metron is completely agnostic to how sensor
> telemetry lands in Kafka.
>
> We also have an Ansible role that will install Snort along with a simple
> mechanism to transport its logs to Kafka.  This is only useful for
> development environments; not a production install.
>
> Using the Ansible role directly may be beyond the knowledge level of
> some.  I only offer this as a guide that you can use to follow along and
> manually install it yourself.
>
> https://github.com/apache/metron/blob/master/metron-
> deployment/roles/snort/tasks/main.yml
>
>
> If you are not familiar with how Ansible roles are defined, just start at
> the main.yml, then follow through each of the other files as they are
> included.  It is pretty readable once you get use to the layout.
>
> On Tue, Oct 17, 2017 at 12:05 PM, Syed Hammad Tahir 
> wrote:
>
>> Ok, Now I get it. Now should I install snort in vagrant ssh in the normal
>> way snort is usually install on a linux distro or do I need to run some
>> special commands again?
>>
>> On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen  wrote:
>>
>>> In the Full Dev environment, Snort is not installed.  We install "Sensor
>>> Stubs" which is just a mechanism that continually replays canned telemetry
>>> logs repetitively to mimic real sensors.  We have to do this because of
>>> resource constraints when running all of Metron on a single VM.  See the
>>> following for more information.
>>>
>>> https://github.com/apache/metron/tree/master/metron-deployme
>>> nt/roles/sensor-stubs
>>>
>>>
>>>
>>> On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 yes,, but when i do snort -v in vagrant ssh console it says snort isnt
 installed where as it can be seen working in metron. Due to that reason I
 am confused because James Sirota said to install snort.

 On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen  wrote:

> From Metron's perspective, Snort is just another sensor.  Snort is
> installed, managed and executed completely independent of Metron itself. 
> As
> with any sensor, you are responsible for getting the telemetry produced by
> Snort into Kafka.  Metron can then consume that telemetry from Kafka and 
> do
> wonderful things with it. :)
>
>
> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> And I am sorry about one confusion but isnt snort builtin into the
>> metron framework? If so then cant we access that snort and do the tasks 
>> you
>> mentioned earlier?
>>
>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> Hi,
>>>
>>> Thanks for the support. Can it be performed both on dumped log and
>>> real time data?
>>> Regards.
>>>
>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota 
>>> wrote:
>>>
 What I mean is that you should install snort, load the appropriate
 Snort rules for your use case, set Snort to log to a directory, and 
 send
 traffic to the network interface where Snort is listening. That will
 produce Snort log files. Then you can push the contents of Snort logs
 either to Kafka using NiFi (preferred) or using Kafka utilities such as
 command line producer. This should be pushed to a Kafka topic called 
 Snort
 where each message is a log line of the Snort file. Does that make 
 sense?

 Thanks,
 James


 11.10.2017, 23:08, "Syed Hammad Tahir" :

 You mean that I must start snort from terminal by doing snort -v
 and then push it to kafka topic? I need to start snort in packet 
 capture
 mode.

 On Tue, Oct 10, 2017 at 9:52 PM, James Sirota 
 wrote:

 Yes, you can use Snort. Metron can consume Snort telemetries out of
 the box. You have to setup Snort on your own and push the output into a
 kafka topic (most likely using NiFi). From there on you can use the 
 output
 of Snort in Metron.


 10.10.2017, 00:48, "Syed Hammad Tahir" :

 Hi,

Re: Snort

[Nick Allen]

No special commands.  Install and configure Snort however you like and get
those logs into a Kafka topic.  Metron is completely agnostic to how sensor
telemetry lands in Kafka.

We also have an Ansible role that will install Snort along with a simple
mechanism to transport its logs to Kafka.  This is only useful for
development environments; not a production install.

Using the Ansible role directly may be beyond the knowledge level of some.
I only offer this as a guide that you can use to follow along and manually
install it yourself.

https://github.com/apache/metron/blob/master/metron-deployment/roles/snort/tasks/main.yml


If you are not familiar with how Ansible roles are defined, just start at
the main.yml, then follow through each of the other files as they are
included.  It is pretty readable once you get use to the layout.

On Tue, Oct 17, 2017 at 12:05 PM, Syed Hammad Tahir 
wrote:

> Ok, Now I get it. Now should I install snort in vagrant ssh in the normal
> way snort is usually install on a linux distro or do I need to run some
> special commands again?
>
> On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen  wrote:
>
>> In the Full Dev environment, Snort is not installed.  We install "Sensor
>> Stubs" which is just a mechanism that continually replays canned telemetry
>> logs repetitively to mimic real sensors.  We have to do this because of
>> resource constraints when running all of Metron on a single VM.  See the
>> following for more information.
>>
>> https://github.com/apache/metron/tree/master/metron-deployme
>> nt/roles/sensor-stubs
>>
>>
>>
>> On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir > > wrote:
>>
>>> yes,, but when i do snort -v in vagrant ssh console it says snort isnt
>>> installed where as it can be seen working in metron. Due to that reason I
>>> am confused because James Sirota said to install snort.
>>>
>>> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen  wrote:
>>>
 From Metron's perspective, Snort is just another sensor.  Snort is
 installed, managed and executed completely independent of Metron itself. As
 with any sensor, you are responsible for getting the telemetry produced by
 Snort into Kafka.  Metron can then consume that telemetry from Kafka and do
 wonderful things with it. :)


 On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> And I am sorry about one confusion but isnt snort builtin into the
> metron framework? If so then cant we access that snort and do the tasks 
> you
> mentioned earlier?
>
> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> Hi,
>>
>> Thanks for the support. Can it be performed both on dumped log and
>> real time data?
>> Regards.
>>
>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota 
>> wrote:
>>
>>> What I mean is that you should install snort, load the appropriate
>>> Snort rules for your use case, set Snort to log to a directory, and send
>>> traffic to the network interface where Snort is listening. That will
>>> produce Snort log files. Then you can push the contents of Snort logs
>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as
>>> command line producer. This should be pushed to a Kafka topic called 
>>> Snort
>>> where each message is a log line of the Snort file. Does that make 
>>> sense?
>>>
>>> Thanks,
>>> James
>>>
>>>
>>> 11.10.2017, 23:08, "Syed Hammad Tahir" :
>>>
>>> You mean that I must start snort from terminal by doing snort -v and
>>> then push it to kafka topic? I need to start snort in packet capture 
>>> mode.
>>>
>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota 
>>> wrote:
>>>
>>> Yes, you can use Snort. Metron can consume Snort telemetries out of
>>> the box. You have to setup Snort on your own and push the output into a
>>> kafka topic (most likely using NiFi). From there on you can use the 
>>> output
>>> of Snort in Metron.
>>>
>>>
>>> 10.10.2017, 00:48, "Syed Hammad Tahir" :
>>>
>>> Hi,
>>>
>>> Can I use snort in packet capture mode with metron? By default it
>>> works in IDS mode only.
>>>
>>> Regards.
>>>
>>>
>>>
>>> ---
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>>
>>>
>>>
>>> ---
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>>
>>
>

>>>
>>
>

Re: event correlation on metron

[Laurens Vets]

Hi Youzha, 

Either check how the snort logs on the full dev installation are
ingested (I believe it's with a script) or check the Apache NiFi project
which makes it very easy to read logs from almost any format and ingest
them to Metron via Kafka. 

On 2017-10-17 08:53, Youzha wrote:

> is it possible to ingest other logs like /var/log/secure for example to be 
> new telemetry on metron? i've seen the metron architecture on the website 
> like picture below. host logs, email, av, etc can be telemetry event buffer 
> on metron. if this possible, could you give me some suggestion how to do it ? 
> 
> On Tue, 17 Oct 2017 at 21.00 Nick Allen  wrote: 
> 
> If you want to look at failed login attempts for each user over time, then 
> the Profiler might be a good solution.  Your profile will depend on the 
> fields available in your telemetry, but it would look something like this, as 
> an example. 
> 
> { 
> 
> "profile": "failed-logins", 
> 
> "foreach": "user.name [1]", 
> 
> "onlyif": "source.type == 'activedirectory' and event.type == 'failed_login'" 
> 
> "init": { "count": 0 }, 
> 
> "update": { "count" : "count + 1" }, 
> 
> "result": "count" 
> 
> } 
> 
> You can find an introduction and more information on using the Profiler 
> below. 
> * 
> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler 
> * https://www.slideshare.net/secret/GFBf2RTXBG35PB 
> 
> Best of luck 
> 
> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul  wrote:
> for example,
> 
> i wanna try to correlate between logs.
> how many times user A have login failed and how many times user A have login 
> succeed. include detail IP, timestamp etc.
> is this possible to do with metron? 
> 
> On 17/10/17 02:56, James Sirota wrote:
> What specifically are you looking to correlate?  Can you talk a little more 
> about your use case?
> 
> 16.10.2017, 02:23, "tkg_cangkul" :
> hi,
> 
> anyone could explain me about event correlation using apache metron?
> does metron support event correlation?
> 
> Pls Advice ---
> Thank you,
> 
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org

 

Links:
--
[1] http://user.name

Re: Snort

[Syed Hammad Tahir]

Ok, Now I get it. Now should I install snort in vagrant ssh in the normal
way snort is usually install on a linux distro or do I need to run some
special commands again?

On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen  wrote:

> In the Full Dev environment, Snort is not installed.  We install "Sensor
> Stubs" which is just a mechanism that continually replays canned telemetry
> logs repetitively to mimic real sensors.  We have to do this because of
> resource constraints when running all of Metron on a single VM.  See the
> following for more information.
>
> https://github.com/apache/metron/tree/master/metron-
> deployment/roles/sensor-stubs
>
>
>
> On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir 
> wrote:
>
>> yes,, but when i do snort -v in vagrant ssh console it says snort isnt
>> installed where as it can be seen working in metron. Due to that reason I
>> am confused because James Sirota said to install snort.
>>
>> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen  wrote:
>>
>>> From Metron's perspective, Snort is just another sensor.  Snort is
>>> installed, managed and executed completely independent of Metron itself. As
>>> with any sensor, you are responsible for getting the telemetry produced by
>>> Snort into Kafka.  Metron can then consume that telemetry from Kafka and do
>>> wonderful things with it. :)
>>>
>>>
>>> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir >> > wrote:
>>>
 And I am sorry about one confusion but isnt snort builtin into the
 metron framework? If so then cant we access that snort and do the tasks you
 mentioned earlier?

 On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> Hi,
>
> Thanks for the support. Can it be performed both on dumped log and
> real time data?
> Regards.
>
> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota 
> wrote:
>
>> What I mean is that you should install snort, load the appropriate
>> Snort rules for your use case, set Snort to log to a directory, and send
>> traffic to the network interface where Snort is listening. That will
>> produce Snort log files. Then you can push the contents of Snort logs
>> either to Kafka using NiFi (preferred) or using Kafka utilities such as
>> command line producer. This should be pushed to a Kafka topic called 
>> Snort
>> where each message is a log line of the Snort file. Does that make sense?
>>
>> Thanks,
>> James
>>
>>
>> 11.10.2017, 23:08, "Syed Hammad Tahir" :
>>
>> You mean that I must start snort from terminal by doing snort -v and
>> then push it to kafka topic? I need to start snort in packet capture 
>> mode.
>>
>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota 
>> wrote:
>>
>> Yes, you can use Snort. Metron can consume Snort telemetries out of
>> the box. You have to setup Snort on your own and push the output into a
>> kafka topic (most likely using NiFi). From there on you can use the 
>> output
>> of Snort in Metron.
>>
>>
>> 10.10.2017, 00:48, "Syed Hammad Tahir" :
>>
>> Hi,
>>
>> Can I use snort in packet capture mode with metron? By default it
>> works in IDS mode only.
>>
>> Regards.
>>
>>
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>
>>
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>
>

>>>
>>
>

Fwd: event correlation on metron

[Youzha]

-- Forwarded message -
From: Youzha 
Date: Tue, 17 Oct 2017 at 22.53
Subject: Re: event correlation on metron
To: 


is it possible to ingest other logs like /var/log/secure for example to be
new telemetry on metron? i’ve seen the metron architecture on the website
like picture below. host logs, email, av, etc can be telemetry event buffer
on metron. if this possible, could you give me some suggestion how to do it
?


On Tue, 17 Oct 2017 at 21.00 Nick Allen  wrote:

> If you want to look at failed login attempts for each user over time, then
> the Profiler might be a good solution.  Your profile will depend on the
> fields available in your telemetry, but it would look something like this,
> as an example.
>
> {
>   "profile": "failed-logins",
>   "foreach": "user.name",
>   "onlyif": "source.type == 'activedirectory' and event.type ==
> 'failed_login'"
>   "init": { "count": 0 },
>   "update": { "count" : "count + 1" },
>   "result": "count"
> }
>
>
> You can find an introduction and more information on using the Profiler
> below.
> *
> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>
> Best of luck
>
> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul 
> wrote:
>
>> for example,
>>
>> i wanna try to correlate between logs.
>> how many times user A have login failed and how many times user A have
>> login succeed. include detail IP, timestamp etc.
>> is this possible to do with metron?
>>
>>
>>
>>
>> On 17/10/17 02:56, James Sirota wrote:
>>
>>> What specifically are you looking to correlate?  Can you talk a little
>>> more about your use case?
>>>
>>> 16.10.2017, 02:23, "tkg_cangkul" :
>>>
 hi,

 anyone could explain me about event correlation using apache metron?
 does metron support event correlation?

 Pls Advice

>>> ---
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>
>>
>

Re: event correlation on metron

[Youzha]

is it possible to ingest other logs like /var/log/secure for example to be
new telemetry on metron? i’ve seen the metron architecture on the website
like picture below. host logs, email, av, etc can be telemetry event buffer
on metron. if this possible, could you give me some suggestion how to do it
?


On Tue, 17 Oct 2017 at 21.00 Nick Allen  wrote:

> If you want to look at failed login attempts for each user over time, then
> the Profiler might be a good solution.  Your profile will depend on the
> fields available in your telemetry, but it would look something like this,
> as an example.
>
> {
>   "profile": "failed-logins",
>   "foreach": "user.name",
>   "onlyif": "source.type == 'activedirectory' and event.type ==
> 'failed_login'"
>   "init": { "count": 0 },
>   "update": { "count" : "count + 1" },
>   "result": "count"
> }
>
>
> You can find an introduction and more information on using the Profiler
> below.
> *
> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>
> Best of luck
>
> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul 
> wrote:
>
>> for example,
>>
>> i wanna try to correlate between logs.
>> how many times user A have login failed and how many times user A have
>> login succeed. include detail IP, timestamp etc.
>> is this possible to do with metron?
>>
>>
>>
>>
>> On 17/10/17 02:56, James Sirota wrote:
>>
>>> What specifically are you looking to correlate?  Can you talk a little
>>> more about your use case?
>>>
>>> 16.10.2017, 02:23, "tkg_cangkul" :
>>>
 hi,

 anyone could explain me about event correlation using apache metron?
 does metron support event correlation?

 Pls Advice

>>> ---
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>
>>
>

Re: Snort

[Nick Allen]

In the Full Dev environment, Snort is not installed.  We install "Sensor
Stubs" which is just a mechanism that continually replays canned telemetry
logs repetitively to mimic real sensors.  We have to do this because of
resource constraints when running all of Metron on a single VM.  See the
following for more information.

https://github.com/apache/metron/tree/master/metron-deployment/roles/sensor-stubs



On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir 
wrote:

> yes,, but when i do snort -v in vagrant ssh console it says snort isnt
> installed where as it can be seen working in metron. Due to that reason I
> am confused because James Sirota said to install snort.
>
> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen  wrote:
>
>> From Metron's perspective, Snort is just another sensor.  Snort is
>> installed, managed and executed completely independent of Metron itself. As
>> with any sensor, you are responsible for getting the telemetry produced by
>> Snort into Kafka.  Metron can then consume that telemetry from Kafka and do
>> wonderful things with it. :)
>>
>>
>> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir 
>> wrote:
>>
>>> And I am sorry about one confusion but isnt snort builtin into the
>>> metron framework? If so then cant we access that snort and do the tasks you
>>> mentioned earlier?
>>>
>>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 Hi,

 Thanks for the support. Can it be performed both on dumped log and real
 time data?
 Regards.

 On Tue, Oct 17, 2017 at 1:02 AM, James Sirota 
 wrote:

> What I mean is that you should install snort, load the appropriate
> Snort rules for your use case, set Snort to log to a directory, and send
> traffic to the network interface where Snort is listening. That will
> produce Snort log files. Then you can push the contents of Snort logs
> either to Kafka using NiFi (preferred) or using Kafka utilities such as
> command line producer. This should be pushed to a Kafka topic called Snort
> where each message is a log line of the Snort file. Does that make sense?
>
> Thanks,
> James
>
>
> 11.10.2017, 23:08, "Syed Hammad Tahir" :
>
> You mean that I must start snort from terminal by doing snort -v and
> then push it to kafka topic? I need to start snort in packet capture mode.
>
> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota 
> wrote:
>
> Yes, you can use Snort. Metron can consume Snort telemetries out of
> the box. You have to setup Snort on your own and push the output into a
> kafka topic (most likely using NiFi). From there on you can use the output
> of Snort in Metron.
>
>
> 10.10.2017, 00:48, "Syed Hammad Tahir" :
>
> Hi,
>
> Can I use snort in packet capture mode with metron? By default it
> works in IDS mode only.
>
> Regards.
>
>
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>
>
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>

>>>
>>
>

Re: Snort

[Syed Hammad Tahir]

yes,, but when i do snort -v in vagrant ssh console it says snort isnt
installed where as it can be seen working in metron. Due to that reason I
am confused because James Sirota said to install snort.

On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen  wrote:

> From Metron's perspective, Snort is just another sensor.  Snort is
> installed, managed and executed completely independent of Metron itself. As
> with any sensor, you are responsible for getting the telemetry produced by
> Snort into Kafka.  Metron can then consume that telemetry from Kafka and do
> wonderful things with it. :)
>
>
> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir 
> wrote:
>
>> And I am sorry about one confusion but isnt snort builtin into the metron
>> framework? If so then cant we access that snort and do the tasks you
>> mentioned earlier?
>>
>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir > > wrote:
>>
>>> Hi,
>>>
>>> Thanks for the support. Can it be performed both on dumped log and real
>>> time data?
>>> Regards.
>>>
>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota 
>>> wrote:
>>>
 What I mean is that you should install snort, load the appropriate
 Snort rules for your use case, set Snort to log to a directory, and send
 traffic to the network interface where Snort is listening. That will
 produce Snort log files. Then you can push the contents of Snort logs
 either to Kafka using NiFi (preferred) or using Kafka utilities such as
 command line producer. This should be pushed to a Kafka topic called Snort
 where each message is a log line of the Snort file. Does that make sense?

 Thanks,
 James


 11.10.2017, 23:08, "Syed Hammad Tahir" :

 You mean that I must start snort from terminal by doing snort -v and
 then push it to kafka topic? I need to start snort in packet capture mode.

 On Tue, Oct 10, 2017 at 9:52 PM, James Sirota 
 wrote:

 Yes, you can use Snort. Metron can consume Snort telemetries out of the
 box. You have to setup Snort on your own and push the output into a kafka
 topic (most likely using NiFi). From there on you can use the output of
 Snort in Metron.


 10.10.2017, 00:48, "Syed Hammad Tahir" :

 Hi,

 Can I use snort in packet capture mode with metron? By default it works
 in IDS mode only.

 Regards.



 ---
 Thank you,

 James Sirota
 PMC- Apache Metron
 jsirota AT apache DOT org




 ---
 Thank you,

 James Sirota
 PMC- Apache Metron
 jsirota AT apache DOT org


>>>
>>
>

Re: Snort

[Nick Allen]

>From Metron's perspective, Snort is just another sensor.  Snort is
installed, managed and executed completely independent of Metron itself. As
with any sensor, you are responsible for getting the telemetry produced by
Snort into Kafka.  Metron can then consume that telemetry from Kafka and do
wonderful things with it. :)


On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir 
wrote:

> And I am sorry about one confusion but isnt snort builtin into the metron
> framework? If so then cant we access that snort and do the tasks you
> mentioned earlier?
>
> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir 
> wrote:
>
>> Hi,
>>
>> Thanks for the support. Can it be performed both on dumped log and real
>> time data?
>> Regards.
>>
>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota  wrote:
>>
>>> What I mean is that you should install snort, load the appropriate Snort
>>> rules for your use case, set Snort to log to a directory, and send traffic
>>> to the network interface where Snort is listening. That will produce Snort
>>> log files. Then you can push the contents of Snort logs either to Kafka
>>> using NiFi (preferred) or using Kafka utilities such as command line
>>> producer. This should be pushed to a Kafka topic called Snort where each
>>> message is a log line of the Snort file. Does that make sense?
>>>
>>> Thanks,
>>> James
>>>
>>>
>>> 11.10.2017, 23:08, "Syed Hammad Tahir" :
>>>
>>> You mean that I must start snort from terminal by doing snort -v and
>>> then push it to kafka topic? I need to start snort in packet capture mode.
>>>
>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota 
>>> wrote:
>>>
>>> Yes, you can use Snort. Metron can consume Snort telemetries out of the
>>> box. You have to setup Snort on your own and push the output into a kafka
>>> topic (most likely using NiFi). From there on you can use the output of
>>> Snort in Metron.
>>>
>>>
>>> 10.10.2017, 00:48, "Syed Hammad Tahir" :
>>>
>>> Hi,
>>>
>>> Can I use snort in packet capture mode with metron? By default it works
>>> in IDS mode only.
>>>
>>> Regards.
>>>
>>>
>>>
>>> ---
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>>
>>>
>>>
>>> ---
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>>
>>
>

Re: event correlation on metron

[Nick Allen]

If you want to look at failed login attempts for each user over time, then
the Profiler might be a good solution.  Your profile will depend on the
fields available in your telemetry, but it would look something like this,
as an example.

{
  "profile": "failed-logins",
  "foreach": "user.name",
  "onlyif": "source.type == 'activedirectory' and event.type ==
'failed_login'"
  "init": { "count": 0 },
  "update": { "count" : "count + 1" },
  "result": "count"
}


You can find an introduction and more information on using the Profiler
below.
*
https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
* https://www.slideshare.net/secret/GFBf2RTXBG35PB

Best of luck

On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul  wrote:

> for example,
>
> i wanna try to correlate between logs.
> how many times user A have login failed and how many times user A have
> login succeed. include detail IP, timestamp etc.
> is this possible to do with metron?
>
>
>
>
> On 17/10/17 02:56, James Sirota wrote:
>
>> What specifically are you looking to correlate?  Can you talk a little
>> more about your use case?
>>
>> 16.10.2017, 02:23, "tkg_cangkul" :
>>
>>> hi,
>>>
>>> anyone could explain me about event correlation using apache metron?
>>> does metron support event correlation?
>>>
>>> Pls Advice
>>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>
>

Re: event correlation on metron

[tkg_cangkul]

for example,

i wanna try to correlate between logs.
how many times user A have login failed and how many times user A have 
login succeed. include detail IP, timestamp etc.

is this possible to do with metron?



On 17/10/17 02:56, James Sirota wrote:

What specifically are you looking to correlate?  Can you talk a little more 
about your use case?

16.10.2017, 02:23, "tkg_cangkul" :

hi,

anyone could explain me about event correlation using apache metron?
does metron support event correlation?

Pls Advice

---
Thank you,

James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

Re: Snort

[Syed Hammad Tahir]

And I am sorry about one confusion but isnt snort builtin into the metron
framework? If so then cant we access that snort and do the tasks you
mentioned earlier?

On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir 
wrote:

> Hi,
>
> Thanks for the support. Can it be performed both on dumped log and real
> time data?
> Regards.
>
> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota  wrote:
>
>> What I mean is that you should install snort, load the appropriate Snort
>> rules for your use case, set Snort to log to a directory, and send traffic
>> to the network interface where Snort is listening. That will produce Snort
>> log files. Then you can push the contents of Snort logs either to Kafka
>> using NiFi (preferred) or using Kafka utilities such as command line
>> producer. This should be pushed to a Kafka topic called Snort where each
>> message is a log line of the Snort file. Does that make sense?
>>
>> Thanks,
>> James
>>
>>
>> 11.10.2017, 23:08, "Syed Hammad Tahir" :
>>
>> You mean that I must start snort from terminal by doing snort -v and then
>> push it to kafka topic? I need to start snort in packet capture mode.
>>
>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota  wrote:
>>
>> Yes, you can use Snort. Metron can consume Snort telemetries out of the
>> box. You have to setup Snort on your own and push the output into a kafka
>> topic (most likely using NiFi). From there on you can use the output of
>> Snort in Metron.
>>
>>
>> 10.10.2017, 00:48, "Syed Hammad Tahir" :
>>
>> Hi,
>>
>> Can I use snort in packet capture mode with metron? By default it works
>> in IDS mode only.
>>
>> Regards.
>>
>>
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>
>>
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>
>