Hi Girish, Based on a few recommendations I restarted the process using the Metron management UI. I am now working through the exercise of scaling down my data input string and Grok format in hopes of isolating the point where the error occurs.
Stay tuned, Frank From: Girish N [mailto:[email protected]] Sent: Thursday, September 14, 2017 11:13 AM To: [email protected] Subject: RE: Grok Parser issues Hi Frank Are you able to overcome Grok Parser issue. If you are, could you please let me know how you were able to do it..Thanks Regards Girish N On 13 Sep 2017 22:35, "Frank Horsfall" <[email protected]<mailto:[email protected]>> wrote: I’ll get back to you once I have it running. Frank From: Girish N [mailto:[email protected]<mailto:[email protected]>] Sent: Wednesday, September 13, 2017 12:26 PM To: [email protected]<mailto:[email protected]> Subject: Re: Grok Parser issues Yes I am also facing the same issues. On 13 Sep 2017 20:06, "Frank Horsfall" <[email protected]<mailto:[email protected]>> wrote: Morning all, Is anyone else seeing this error? After successfully going through the telemetry tutorial with squid (https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating a new telemetry based on a data set I wish to use. Test data Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80 Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80 Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80 Grok statement blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port} I tested the pattern at the Grok site http://grokconstructor.appspot.com/do/match#result [cid:[email protected]] Added the pattern to hdfs [hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port} Dump of zookeeper PARSER Config: blah { "parserClassName": "org.apache.metron.parsers.GrokParser", "sensorTopic": "blah", "parserConfig": { "grokPath": "/apps/metron/patterns/dionaea", "patternLabel": "blah_DELIMITED", "timestampField": "timestamp" } } INDEXING Config: blah { "elasticsearch": { "index": "blah", "batchSize": 5, "enabled": true }, "hdfs": { "index": "blah", "batchSize": 5, "enabled" : true }, "solr": { "index": "blah", "batchSize": 5, "enabled" : false } } ENRICHMENT Config: blah { "enrichment" : { "fieldMap": { "geo": ["ip_dst_addr", "ip_src_addr"], "host": ["host"] } }, "threatIntel": { "fieldMap": { "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"] }, "fieldToTypeMap": { "ip_src_addr" : ["malicious_ip"], "ip_dst_addr" : ["malicious_ip"] } } } Nifi is set up and passes correctly. But when I get to the parserBolt an error occurs. java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:152) ... 12 more Any ideas? Kindest regards, Frank
