Hello Frank, I had same problem some weeks ago. For me first thing is to check if you don’t have an extra space at end of your grok parser line ?
Another thing is to follow same tutorial but use Metron GUI insteed of command Line. Lehuédé Sebastien > Le 13 sept. 2017 à 16:36, Frank Horsfall <[email protected]> a > écrit : > > Morning all, > > Is anyone else seeing this error? > > > After successfully going through the telemetry tutorial with squid > (https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source > ) I started the exercise of creating a new telemetry based on a data set I > wish to use. > > Test data > > Jul 28 00:13:24 device1 devicelogger.py: > connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 > Jul 28 00:13:44 device1 devicelogger.py: > connection|5289|accept|tcp|httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80 > Jul 28 00:13:44 device1 devicelogger.py: > connection|5288|accept|tcp|httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80 > Jul 28 00:59:49 device1 devicelogger.py: > connection|5296|accept|tcp|httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80 > > Grok statement > > blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} > %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} > %{WORD:script}.%{WORD:extension}: > %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port} > > > I tested the pattern at the Grok site > http://grokconstructor.appspot.com/do/match#result > > <image003.jpg> > > Added the pattern to hdfs > > [hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah > blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} > %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} > %{WORD:script}.%{WORD:extension}: > %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port} > > > Dump of zookeeper > > PARSER Config: blah > { > "parserClassName": "org.apache.metron.parsers.GrokParser", > "sensorTopic": "blah", > "parserConfig": { > "grokPath": "/apps/metron/patterns/dionaea", > "patternLabel": "blah_DELIMITED", > "timestampField": "timestamp" > } > } > > > INDEXING Config: blah > { > "elasticsearch": { > "index": "blah", > "batchSize": 5, > "enabled": true > }, > "hdfs": { > "index": "blah", > "batchSize": 5, > "enabled" : true > }, > "solr": { > "index": "blah", > "batchSize": 5, > "enabled" : false > } > } > > ENRICHMENT Config: blah > { > "enrichment" : { > "fieldMap": > { > "geo": ["ip_dst_addr", "ip_src_addr"], > "host": ["host"] > } > }, > "threatIntel": { > "fieldMap": > { > "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"] > }, > "fieldToTypeMap": > { > "ip_src_addr" : ["malicious_ip"], > "ip_dst_addr" : ["malicious_ip"] > } > } > } > > > Nifi is set up and passes correctly. But when I get to the parserBolt an > error occurs. > > java.lang.IllegalStateException: Grok parser Error: Grok statement produced a > null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: > connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 > and the parsed message was: {} . Check the pattern at: > /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py: > connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 at > org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at > org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) > at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:133) at > org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) > at > org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) > at > org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) > at > org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) > at > org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) > at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at > clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Grok statement produced a null > message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: > connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 > and the parsed message was: {} . Check the pattern at: > /apps/metron/patterns/dionaea at > org.apache.metron.parsers.GrokParser.parse(GrokParser.java:152) ... 12 more > > > Any ideas? > > Kindest regards, > Frank > > > > >
