Do you mean this one on port 4200? [cid:[email protected]]
From: Ryan Merriman [mailto:[email protected]] Sent: Wednesday, September 13, 2017 10:53 AM To: [email protected] Subject: Re: Grok Parser issues You're better off using our management UI, it runs the same code that the parser topology does. I would start small with just a couple expressions (something like "blah_DELIMITED %{MONTH:month}") and ensure you're at least getting back the month. Then you can incrementally add more on until you find out where your problem is. Ryan On Wed, Sep 13, 2017 at 9:36 AM, Frank Horsfall <[email protected]<mailto:[email protected]>> wrote: Morning all, Is anyone else seeing this error? After successfully going through the telemetry tutorial with squid (https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating a new telemetry based on a data set I wish to use. Test data Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80 Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80 Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80 Grok statement blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port} I tested the pattern at the Grok site http://grokconstructor.appspot.com/do/match#result [cid:[email protected]] Added the pattern to hdfs [hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port} Dump of zookeeper PARSER Config: blah { "parserClassName": "org.apache.metron.parsers.GrokParser", "sensorTopic": "blah", "parserConfig": { "grokPath": "/apps/metron/patterns/dionaea", "patternLabel": "blah_DELIMITED", "timestampField": "timestamp" } } INDEXING Config: blah { "elasticsearch": { "index": "blah", "batchSize": 5, "enabled": true }, "hdfs": { "index": "blah", "batchSize": 5, "enabled" : true }, "solr": { "index": "blah", "batchSize": 5, "enabled" : false } } ENRICHMENT Config: blah { "enrichment" : { "fieldMap": { "geo": ["ip_dst_addr", "ip_src_addr"], "host": ["host"] } }, "threatIntel": { "fieldMap": { "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"] }, "fieldToTypeMap": { "ip_src_addr" : ["malicious_ip"], "ip_dst_addr" : ["malicious_ip"] } } } Nifi is set up and passes correctly. But when I get to the parserBolt an error occurs. java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:152) ... 12 more Any ideas? Kindest regards, Frank
