Morning all, Is anyone else seeing this error?
After successfully going through the telemetry tutorial with squid (https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating a new telemetry based on a data set I wish to use. Test data Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80 Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80 Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80 Grok statement blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port} I tested the pattern at the Grok site http://grokconstructor.appspot.com/do/match#result [cid:[email protected]] Added the pattern to hdfs [hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port} Dump of zookeeper PARSER Config: blah { "parserClassName": "org.apache.metron.parsers.GrokParser", "sensorTopic": "blah", "parserConfig": { "grokPath": "/apps/metron/patterns/dionaea", "patternLabel": "blah_DELIMITED", "timestampField": "timestamp" } } INDEXING Config: blah { "elasticsearch": { "index": "blah", "batchSize": 5, "enabled": true }, "hdfs": { "index": "blah", "batchSize": 5, "enabled" : true }, "solr": { "index": "blah", "batchSize": 5, "enabled" : false } } ENRICHMENT Config: blah { "enrichment" : { "fieldMap": { "geo": ["ip_dst_addr", "ip_src_addr"], "host": ["host"] } }, "threatIntel": { "fieldMap": { "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"] }, "fieldToTypeMap": { "ip_src_addr" : ["malicious_ip"], "ip_dst_addr" : ["malicious_ip"] } } } Nifi is set up and passes correctly. But when I get to the parserBolt an error occurs. java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:152) ... 12 more Any ideas? Kindest regards, Frank
