Hey Girish, Can you validate using http://grokconstructor.appspot.com/do/match that you can parse the sample message using the pattern? I tried with your example and it did not work.
-Anand On 9/6/17, 4:06 PM, "Girish N" <[email protected]> wrote: >Thanks for your response Otto Fowler, > >I tried with the below config. Still the same exception. Kindly let me know >if anything else has to be changed. > >{ >"parserClassName": "org.apache.metron.parsers.GrokParser", >"sensorTopic": "log", >"parserConfig": { >"grokPath": "/patterns/log", >"patternLabel": "SYS_DELIMITED" >}} > >java.lang.IllegalStateException: Unhandled bulk errors in response: >{java.lang.IllegalArgumentException: Topic cannot be null=[source: >parserBolt:4, stream: error, id: {}, >[{"exception":"java.lang.IllegalStateException: >Grok parser Error: Grok statement produced a null message. Original message >was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. >and the parsed message was: {} . Check the pattern at: \/patterns\/log on >Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS >Scheduler.","failed_sensor_type":"log","stack":"java.lang.IllegalStateException: >Grok parser Error: Grok statement produced a null message. Original message >was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. >and the parsed message was: {} > > >Regards >Girish N > > >On Wed, Sep 6, 2017 at 3:45 PM, Otto Fowler <[email protected]> wrote: > >> Change the grokPath to match below and try. >> >> { >> "parserClassName": "org.apache.metron.parsers.GrokParser", >> "sensorTopic": "log", >> "parserConfig": { >> "grokPath": "/patterns/log", >> "patternLabel": "SYS_DELIMITED" >> }} >> >> >> This path is not an absolute path. >> >> On September 6, 2017 at 05:16:19, Girish N (giri.narasimha.murthy@gmail. >> com) wrote: >> >> Hi, >> >> I am trying to parse the syslog I am getting below exceptions. Kindly help >> to resolve the issue. Thanks >> >> >> Sample Syslog- >> Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend. >> >> >> 1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log >> SYS_DELIMITED >> <%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )? >> %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg} >> >> 2. Created a Parser config log.json >> { >> "parserClassName": "org.apache.metron.parsers.GrokParser", >> "sensorTopic": "log", >> "parserConfig": { >> "grokPath": "usr/metron/0.4.0/patterns/log", >> "patternLabel": "SYS_DELIMITED" >> }} >> >> 3. Created a indexing config log.json >> { >> "hdfs" : { >> "index": "log", >> "batchSize": 5, >> "enabled" : true >> }, >> "elasticsearch" : { >> "index": "log", >> "batchSize": 5, >> "enabled" : true >> }, >> "solr" : { >> "index": "log", >> "batchSize": 5, >> "enabled" : true >> } >> } >> >> 4. Pushed and Dumped using >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i >> /usr/metron/0.4.0/config/zookeeper >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP >> >> 5. Then started the parser topology , getting the below exceptions in log >> worker-artifacts ( >> usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700) >> >> java.lang.IllegalStateException: Unhandled bulk errors in response: >> {java.lang.IllegalArgumentException: Topic cannot be null=[source: >> parserBolt:4, stream: error, id: {}, >> [{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok >> statement produced a null message. Original message was: Sep 6 12:41:02 >> exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed >> message was: {} . Check the pattern at: \/patterns\/log on Sep 6 12:41:02 >> exza-ThinkPad-X240 systemd[1]: Started CUPS >> Scheduler.","failed_sensor_type":"log","stack":"java.lang.IllegalStateException: >> >> Grok parser Error: Grok statement produced a null message. Original >> message >> was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. >> and the parsed message was: {} >> >> 6. For the enrichment topology, getting the below exception. >> 2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The configuration >> request.required.acks = 1 was supplied but isn't a known config. >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version : >> 0.10.0.1 >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId : >> a7a17cdec9eaa6c5 >> 2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt >> enrichmentErrorOutputBolt:(1) >> 2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using >> System classloader >> 2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt >> threatIntelSplitBolt:(14) >> 2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt >> enrichmentSplitBolt:(3) >> 2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt >> hostEnrichmentBolt:(5) >> 2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt >> enrichmentJoinBolt:(2) >> 2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR] >> org.apache.metron.jackson.core.JsonParseException: Unrecognized token >> 'enrichments': was expecting ('true', 'false' or 'null') >> at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23] >> at >> org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.java:1586) >> >> ~[stormjar.jar:?] >> at >> org.apache.metron.jackson.core.base.ParserMinimalBase._ >> reportError(ParserMinimalBase.java:521) >> ~[stormjar.jar:?] >> at >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._ >> reportInvalidToken(UTF8StreamJsonParser.java:3466) >> ~[stormjar.jar:?] >> at >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._ >> handleUnexpectedValue(UTF8StreamJsonParser.java:2625) >> ~[stormjar.jar:?] >> at >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._ >> nextTokenNotInObject(UTF8StreamJsonParser.java:854) >> ~[stormjar.jar:?] >> at >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:748) >> >> ~[stormjar.jar:?] >> at >> org.apache.metron.jackson.databind.ObjectMapper._ >> initForReading(ObjectMapper.java:3847) >> ~[stormjar.jar:?] >> at >> org.apache.metron.jackson.databind.ObjectMapper._ >> readMapAndClose(ObjectMapper.java:3792) >> ~[stormjar.jar:?] >> at >> org.apache.metron.jackson.databind.ObjectMapper. >> readValue(ObjectMapper.java:2867) >> ~[stormjar.jar:?] >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55) >> ~[stormjar.jar:?] >> >> 7. For the Indexing topology , getting below exception >> com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException: >> Unrecognized token 'indexing': was expecting ('true', 'false' or 'null') >> at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17] >> at >> com.fasterxml.jackson.core.metron.elasticsearch. >> JsonParser._constructError(JsonParser.java:1581) >> ~[stormjar.jar:?] >> at >> com.fasterxml.jackson.core.metron.elasticsearch.base.ParserMinimalBase._ >> reportError(ParserMinimalBase.java:533) >> ~[stormjar.jar:?] >> at >> com.fasterxml.jackson.core.metron.elasticsearch.json. >> UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3451) >> ~[stormjar.jar:?] >> at >> com.fasterxml.jackson.core.metron.elasticsearch.json. >> UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2610) >> >> ~[stormjar.jar:?] >> at >> com.fasterxml.jackson.core.metron.elasticsearch.json. >> UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:841) >> ~[stormjar.jar:?] >> at >> com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser. >> nextToken(UTF8StreamJsonParser.java:737) >> ~[stormjar.jar:?] >> at >> com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847) >> >> ~[stormjar.jar:?] >> at >> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792) >> >> ~[stormjar.jar:?] >> at >> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2874) >> >> ~[stormjar.jar:?] >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41) >> ~[stormjar.jar:?] >> at >> org.apache.metron.common.configuration.IndexingConfigurations. >> updateSensorIndexingConfig(IndexingConfigurations.java:52) >> ~[stormjar.jar:?] >> >> >> Kindly help to resolve the issue. >> >> Regards >> Girish N >> >>
