Re: SysLog using CEF Parser (RSysLogs)

2018-01-22 Thread Farrukh Naveed Anjum 
Any suggestion how to fix that ?

On Mon, Jan 22, 2018 at 9:01 PM, Farrukh Naveed Anjum <
anjum.farr...@gmail.com> wrote:

> Hi Simon,
>
> Thanks for replying yes, these are indexing bolt errors. I am basically
> trying to forward RSyslog via Nifi. It comes down all the way till indexing
> bolts causes error.
>
> My purpose of using Generic CEF Parser is so that it accumolate SysLog ? I
> did not give him any format, just created a CEF Parsers in Metron
> Management UI. Do I need to give some kind of pattern too ? Or it can
> figure out default syslog pattern ? Kindly guide
>
> By the way following is the indexing bolt error
>
>
>   at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) 
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2018-01-16 02:34:16.543 o.a.s.d.executor [ERROR]
> java.lang.Exception: WARNING: Default and (likely) unoptimized writer config 
> used for hdfs writer and sensor profiler
>   at 
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
>  [stormjar.jar:?]
>   at 
> org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) 
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2018-01-16 02:34:16.543 o.a.s.d.executor [ERROR]
> java.lang.Exception: WARNING: Default and (likely) unoptimized writer config 
> used for elasticsearch writer and sensor profiler
>   at 
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
>  [stormjar.jar:?]
>   at 
> org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) 
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2018-01-16 02:34:16.547 o.a.s.d.executor [ERROR]
> java.lang.Exception: WARNING: Default and (likely) unoptimized writer config 
> used for hdfs writer and sensor profiler
>   at 
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
>  [stormjar.jar:?]
>   at 
> org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
>

Re: Some Metron Alerts UI questions

2018-01-22 Thread Simon Elliston Ball 
Hi Laurens, 

A few quick answers inline…

Simon

> On 20 Jan 2018, at 00:37, Laurens Vets  wrote:
> 
> Hi list,
> 
> I have some general Alerts UI questions/comments/remarks, I hope you don't 
> mind :) I'm using the UI that's part of Metron 0.4.2. These apply to my 
> specific use case, so I might be completely wrong in how I use the UI…

Comment and feedback are always welcome!

> 
> - When you're talking about 'alerts', from what I can see in the UI, that's 
> synonymous with just events in elasticsearch right? Wouldn't it make more 
> sense to treat alerts as events where "is_alert" == True?
> 

At present the search does not exclude non-alerts… it’s maybe a little odd to 
call it the alerts view right now, but right now it’s the only way to see 
everything, so this should probably separate out into an ‘everything’ hunting 
focused view and a alerts only view.

The reasons I kinda like the current approach is that it’s good for picking up 
things that have become alerts because they’re in threat intel for example, 
along with things clustered against them by something like the new TLSH 
functions, which makes it easier to combine known alerts with un-detected 
events in a meta alert.

> - It seems that everything I do in the UI is only stored locally? See 
> https://github.com/apache/metron/tree/master/metron-interface/metron-alerts. 
> Can this made persistent for multiple people?

Yep. A lot of the preferences, saved searched, column layouts etc, are stored 
in local storage by the browser right now. We need a REST endpoint and to 
figure out how to store them (against user / against a group / global??? 
thoughts?) server side. A lot of the mechanism to do that is in, it’s just not 
quite done done because of those open questions I expect. 

> 
> - How can I change the content "Filters" on the left of the UI?

You wait for https://github.com/apache/metron/pull/853 
 to land. 

> 
> - How do I create a MetaAlert?

You can create a meta-alert from a grouped set of alerts, use the grouping 
buttons at the top and you’ll find a merge alert. Slightly odd process at the 
moment true, but a button to create a meta-alert from all the selected, or all 
the visible alerts on the results page might be a good addition, what do you 
think?

Very quick video of the current method here: https://youtu.be/JkFeNKTOd38

> 
> - What's the plan regarding notifying someone when alerts triggers?

Currently there is no external notification, but the answer here would likely 
be to consume the indexing topic in kafka and integrate to an enterprise alarm 
or monitoring system (alerting and alarms is a massive topic which probably 
deserves its own project beyond metron and I’ve seen people use all sorts of 
things for this, usually some big enterprisey thing mandated by IT).

Re: SysLog using CEF Parser (RSysLogs)

2018-01-22 Thread Otto Fowler 
If it reaches the Indexing topology it is not a Parser problem, in almost
all cases.



On January 22, 2018 at 03:24:35, Farrukh Naveed Anjum (
anjum.farr...@gmail.com) wrote:

Yes its Strom Indexing Bolt that is halting it. Any one working on CEF
Parser (Can Syslog work with it like RSyslog). We are stuck at that point.

Please see the above error and suggest

On Mon, Jan 22, 2018 at 1:10 PM, Gaurav Bapat  wrote:

> Hi,
>
> Even I am stuck with the same, and dont know how to solve the issue.
>
> Looks like this is a parsing error
>
> On 22 January 2018 at 13:00, Farrukh Naveed Anjum  > wrote:
>
>> Hi,
>>
>> I am trying to Ingest syslog using CEF Parser it is not creating any
>> Elastic Search Index based on.
>>
>> Any suggestion how can I achieve it ?
>>
>>
>>
>>
>> --
>> With Regards
>> Farrukh Naveed Anjum
>>
>
>


--
With Regards
Farrukh Naveed Anjum

Re: SysLog using CEF Parser (RSysLogs)

2018-01-22 Thread Simon Elliston Ball 
Are there any errors in the logs for the indexing bolt? I would expect the 
errors are probably at the elastic ingest point, and probably caused by an 
incorrect elastic template for the CEF data. 

Simon

> On 22 Jan 2018, at 08:24, Farrukh Naveed Anjum  
> wrote:
> 
> Yes its Strom Indexing Bolt that is halting it. Any one working on CEF Parser 
> (Can Syslog work with it like RSyslog). We are stuck at that point.
> 
> Please see the above error and suggest
> 
> On Mon, Jan 22, 2018 at 1:10 PM, Gaurav Bapat  > wrote:
> Hi,
> 
> Even I am stuck with the same, and dont know how to solve the issue.
> 
> Looks like this is a parsing error
> 
> On 22 January 2018 at 13:00, Farrukh Naveed Anjum  > wrote:
> Hi,
> 
> I am trying to Ingest syslog using CEF Parser it is not creating any Elastic 
> Search Index based on. 
> 
> Any suggestion how can I achieve it ?
> 
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum
> 
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: SysLog using CEF Parser (RSysLogs)

2018-01-22 Thread Farrukh Naveed Anjum 
Yes its Strom Indexing Bolt that is halting it. Any one working on CEF
Parser (Can Syslog work with it like RSyslog). We are stuck at that point.

Please see the above error and suggest

On Mon, Jan 22, 2018 at 1:10 PM, Gaurav Bapat  wrote:

> Hi,
>
> Even I am stuck with the same, and dont know how to solve the issue.
>
> Looks like this is a parsing error
>
> On 22 January 2018 at 13:00, Farrukh Naveed Anjum  > wrote:
>
>> Hi,
>>
>> I am trying to Ingest syslog using CEF Parser it is not creating any
>> Elastic Search Index based on.
>>
>> Any suggestion how can I achieve it ?
>>
>>
>>
>>
>> --
>> With Regards
>> Farrukh Naveed Anjum
>>
>
>


-- 
With Regards
Farrukh Naveed Anjum