Re: SysLog using CEF Parser (RSysLogs)

2018-01-22 Thread Farrukh Naveed Anjum 
Any suggestion how to fix that ?

On Mon, Jan 22, 2018 at 9:01 PM, Farrukh Naveed Anjum <
anjum.farr...@gmail.com> wrote:

> Hi Simon,
>
> Thanks for replying yes, these are indexing bolt errors. I am basically
> trying to forward RSyslog via Nifi. It comes down all the way till indexing
> bolts causes error.
>
> My purpose of using Generic CEF Parser is so that it accumolate SysLog ? I
> did not give him any format, just created a CEF Parsers in Metron
> Management UI. Do I need to give some kind of pattern too ? Or it can
> figure out default syslog pattern ? Kindly guide
>
> By the way following is the indexing bolt error
>
>
>   at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) 
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2018-01-16 02:34:16.543 o.a.s.d.executor [ERROR]
> java.lang.Exception: WARNING: Default and (likely) unoptimized writer config 
> used for hdfs writer and sensor profiler
>   at 
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
>  [stormjar.jar:?]
>   at 
> org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) 
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2018-01-16 02:34:16.543 o.a.s.d.executor [ERROR]
> java.lang.Exception: WARNING: Default and (likely) unoptimized writer config 
> used for elasticsearch writer and sensor profiler
>   at 
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
>  [stormjar.jar:?]
>   at 
> org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) 
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2018-01-16 02:34:16.547 o.a.s.d.executor [ERROR]
> java.lang.Exception: WARNING: Default and (likely) unoptimized writer config 
> used for hdfs writer and sensor profiler
>   at 
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
>  [stormjar.jar:?]
>   at 
> org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
>

Re: SysLog using CEF Parser (RSysLogs)

2018-01-22 Thread Otto Fowler 
If it reaches the Indexing topology it is not a Parser problem, in almost
all cases.



On January 22, 2018 at 03:24:35, Farrukh Naveed Anjum (
anjum.farr...@gmail.com) wrote:

Yes its Strom Indexing Bolt that is halting it. Any one working on CEF
Parser (Can Syslog work with it like RSyslog). We are stuck at that point.

Please see the above error and suggest

On Mon, Jan 22, 2018 at 1:10 PM, Gaurav Bapat  wrote:

> Hi,
>
> Even I am stuck with the same, and dont know how to solve the issue.
>
> Looks like this is a parsing error
>
> On 22 January 2018 at 13:00, Farrukh Naveed Anjum  > wrote:
>
>> Hi,
>>
>> I am trying to Ingest syslog using CEF Parser it is not creating any
>> Elastic Search Index based on.
>>
>> Any suggestion how can I achieve it ?
>>
>>
>>
>>
>> --
>> With Regards
>> Farrukh Naveed Anjum
>>
>
>


--
With Regards
Farrukh Naveed Anjum

Re: SysLog using CEF Parser (RSysLogs)

2018-01-22 Thread Simon Elliston Ball 
Are there any errors in the logs for the indexing bolt? I would expect the 
errors are probably at the elastic ingest point, and probably caused by an 
incorrect elastic template for the CEF data. 

Simon

> On 22 Jan 2018, at 08:24, Farrukh Naveed Anjum  
> wrote:
> 
> Yes its Strom Indexing Bolt that is halting it. Any one working on CEF Parser 
> (Can Syslog work with it like RSyslog). We are stuck at that point.
> 
> Please see the above error and suggest
> 
> On Mon, Jan 22, 2018 at 1:10 PM, Gaurav Bapat  > wrote:
> Hi,
> 
> Even I am stuck with the same, and dont know how to solve the issue.
> 
> Looks like this is a parsing error
> 
> On 22 January 2018 at 13:00, Farrukh Naveed Anjum  > wrote:
> Hi,
> 
> I am trying to Ingest syslog using CEF Parser it is not creating any Elastic 
> Search Index based on. 
> 
> Any suggestion how can I achieve it ?
> 
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum
> 
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: SysLog using CEF Parser (RSysLogs)

2018-01-22 Thread Farrukh Naveed Anjum 
Yes its Strom Indexing Bolt that is halting it. Any one working on CEF
Parser (Can Syslog work with it like RSyslog). We are stuck at that point.

Please see the above error and suggest

On Mon, Jan 22, 2018 at 1:10 PM, Gaurav Bapat  wrote:

> Hi,
>
> Even I am stuck with the same, and dont know how to solve the issue.
>
> Looks like this is a parsing error
>
> On 22 January 2018 at 13:00, Farrukh Naveed Anjum  > wrote:
>
>> Hi,
>>
>> I am trying to Ingest syslog using CEF Parser it is not creating any
>> Elastic Search Index based on.
>>
>> Any suggestion how can I achieve it ?
>>
>>
>>
>>
>> --
>> With Regards
>> Farrukh Naveed Anjum
>>
>
>


-- 
With Regards
Farrukh Naveed Anjum