Yes On Wed, Sep 13, 2017 at 10:02 AM, Frank Horsfall < frankhorsf...@cunet.carleton.ca> wrote:
> Do you mean this one on port 4200? > > > > > > *From:* Ryan Merriman [mailto:merrim...@gmail.com] > *Sent:* Wednesday, September 13, 2017 10:53 AM > *To:* user@metron.apache.org > *Subject:* Re: Grok Parser issues > > > > You're better off using our management UI, it runs the same code that the > parser topology does. I would start small with just a couple expressions > (something like "blah_DELIMITED %{MONTH:month}") and ensure you're at > least getting back the month. Then you can incrementally add more on until > you find out where your problem is. > > > > Ryan > > > > On Wed, Sep 13, 2017 at 9:36 AM, Frank Horsfall < > frankhorsf...@cunet.carleton.ca> wrote: > > Morning all, > > > > Is anyone else seeing this error? > > > > > > After successfully going through the telemetry tutorial with squid ( > https://cwiki.apache.org/confluence/display/METRON/ > Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating > a new telemetry based on a data set I wish to use. > > > > Test data > > > > Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp| > httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 > > Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp| > httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80 > > Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp| > httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80 > > Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp| > httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80 > > > > Grok statement > > > > blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} > %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} > %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER: > number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\| > %{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\ > |%{IP:ip_dst_addr}\|%{NUMBER:dst_port} > > > > > > I tested the pattern at the Grok site > > http://grokconstructor.appspot.com/do/match#result > > > > > > Added the pattern to hdfs > > > > [hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah > > blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} > %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} > %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER: > number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\| > %{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\ > |%{IP:ip_dst_addr}\|%{NUMBER:dst_port} > > > > > > *Dump of zookeeper* > > > > PARSER Config: blah > > { > > "parserClassName": "org.apache.metron.parsers.GrokParser", > > "sensorTopic": "blah", > > "parserConfig": { > > "grokPath": "/apps/metron/patterns/dionaea", > > "patternLabel": "blah_DELIMITED", > > "timestampField": "timestamp" > > } > > } > > > > > > INDEXING Config: blah > > { > > "elasticsearch": { > > "index": "blah", > > "batchSize": 5, > > "enabled": true > > }, > > "hdfs": { > > "index": "blah", > > "batchSize": 5, > > "enabled" : true > > }, > > "solr": { > > "index": "blah", > > "batchSize": 5, > > "enabled" : false > > } > > } > > > > ENRICHMENT Config: blah > > { > > "enrichment" : { > > "fieldMap": > > { > > "geo": ["ip_dst_addr", "ip_src_addr"], > > "host": ["host"] > > } > > }, > > "threatIntel": { > > "fieldMap": > > { > > "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"] > > }, > > "fieldToTypeMap": > > { > > "ip_src_addr" : ["malicious_ip"], > > "ip_dst_addr" : ["malicious_ip"] > > } > > } > > } > > > > > > *Nifi is set up and passes correctly. But when I get to the parserBolt > an error occurs.* > > > > java.lang.IllegalStateException: Grok parser Error: Grok statement > produced a null message. Original message was: Jul 28 00:13:24 device1 > devicelogger.py: > connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 > and the parsed message was: {} . Check the pattern at: > /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py: > connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 > at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at > org.apache.metron.parsers.interfaces.MessageParser. > parseOptional(MessageParser.java:45) at org.apache.metron.parsers. > bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon. > executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at > org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) > at > org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) > at > org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) > at > org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) > at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at > clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Grok statement produced a null > message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: > connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 > and the parsed message was: {} . Check the pattern at: > /apps/metron/patterns/dionaea at org.apache.metron.parsers. > GrokParser.parse(GrokParser.java:152) ... 12 more > > > > > > Any ideas? > > > > Kindest regards, > > Frank > > > > > > > > > > > > >
