http://grokconstructor.appspot.com/do/match#result mentions a lot of patterns.
Maybe open a jira? On September 6, 2017 at 07:08:13, ed d ([email protected]) wrote: Does anyone know of a solid repository of Grok patterns that match standard traffic patterns, like Cisco (non-ASA), JunOS, McAfee (all flavors), syslog, CheckPoint, etc? Maybe a repository sectioned into tech, like Firewalls, Endpoints, Network, VPN, and other security tools? Is there a place in Metron we can start dumping some Grok filters? I know there are many parsers already built, but it would be nice to have another method in case the parser does not fit exactly, or we need a non parser solution. ------------------------------ *From:* Girish N <[email protected]> *Sent:* Wednesday, September 6, 2017 7:02 AM *To:* [email protected] *Cc:* Otto Fowler *Subject:* Re: Exceptions in Parser, enrichments and indexing topology Hello, Grok pattern was wrong, changed it as below and tested and got a match. Sample Log - Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend. Grok pattern 1 - %{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )? %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg} Grok pattern 2 - %{SYSLOGLINE} Tried parsing with both the grok patterns, but still facing the same exceptions. Regards Girish N On Wed, Sep 6, 2017 at 4:17 PM, Anand Subramanian < [email protected]> wrote: > Hey Girish, > > Can you validate using http://grokconstructor.appspot.com/do/match that Test grok patterns - Grok Constructor <http://grokconstructor.appspot.com/do/match> grokconstructor.appspot.com This tries to parse a set of given logfile lines with a given grok regular expression (based on Oniguruma regular expressions) and prints the matches for ... > you can parse the sample message using the pattern? I tried with your > example and it did not work. > > -Anand > > > > On 9/6/17, 4:06 PM, "Girish N" <[email protected]> wrote: > > >Thanks for your response Otto Fowler, > > > >I tried with the below config. Still the same exception. Kindly let me > know > >if anything else has to be changed. > > > >{ > >"parserClassName": "org.apache.metron.parsers.GrokParser", > >"sensorTopic": "log", > >"parserConfig": { > >"grokPath": "/patterns/log", > >"patternLabel": "SYS_DELIMITED" > >}} > > > >java.lang.IllegalStateException: Unhandled bulk errors in response: > >{java.lang.IllegalArgumentException: Topic cannot be null=[source: > >parserBolt:4, stream: error, id: {}, > >[{"exception":"java.lang.IllegalStateException: > >Grok parser Error: Grok statement produced a null message. Original > message > >was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS > Scheduler. > >and the parsed message was: {} . Check the pattern at: \/patterns\/log on > >Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS > >Scheduler.","failed_sensor_type":"log","stack":"java. > lang.IllegalStateException: > >Grok parser Error: Grok statement produced a null message. Original > message > >was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS > Scheduler. > >and the parsed message was: {} > > > > > >Regards > >Girish N > > > > > >On Wed, Sep 6, 2017 at 3:45 PM, Otto Fowler <[email protected]> > wrote: > > > >> Change the grokPath to match below and try. > >> > >> { > >> "parserClassName": "org.apache.metron.parsers.GrokParser", > >> "sensorTopic": "log", > >> "parserConfig": { > >> "grokPath": "/patterns/log", > >> "patternLabel": "SYS_DELIMITED" > >> }} > >> > >> > >> This path is not an absolute path. > >> > >> On September 6, 2017 at 05:16:19, Girish N (giri.narasimha.murthy@gmail. > >> com) wrote: > >> > >> Hi, > >> > >> I am trying to parse the syslog I am getting below exceptions. Kindly > help > >> to resolve the issue. Thanks > >> > >> > >> Sample Syslog- > >> Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend. > >> > >> > >> 1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log > >> SYS_DELIMITED > >> <%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} > )? > >> %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg} > >> > >> 2. Created a Parser config log.json > >> { > >> "parserClassName": "org.apache.metron.parsers.GrokParser", > >> "sensorTopic": "log", > >> "parserConfig": { > >> "grokPath": "usr/metron/0.4.0/patterns/log", > >> "patternLabel": "SYS_DELIMITED" > >> }} > >> > >> 3. Created a indexing config log.json > >> { > >> "hdfs" : { > >> "index": "log", > >> "batchSize": 5, > >> "enabled" : true > >> }, > >> "elasticsearch" : { > >> "index": "log", > >> "batchSize": 5, > >> "enabled" : true > >> }, > >> "solr" : { > >> "index": "log", > >> "batchSize": 5, > >> "enabled" : true > >> } > >> } > >> > >> 4. Pushed and Dumped using > >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i > >> /usr/metron/0.4.0/config/zookeeper > >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP > >> > >> 5. Then started the parser topology , getting the below exceptions in > log > >> worker-artifacts ( > >> usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700) > >> > >> java.lang.IllegalStateException: Unhandled bulk errors in response: > >> {java.lang.IllegalArgumentException: Topic cannot be null=[source: > >> parserBolt:4, stream: error, id: {}, > >> [{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok > >> statement produced a null message. Original message was: Sep 6 12:41:02 > >> exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed > >> message was: {} . Check the pattern at: \/patterns\/log on Sep 6 > 12:41:02 > >> exza-ThinkPad-X240 systemd[1]: Started CUPS > >> Scheduler.","failed_sensor_type":"log","stack":"java. > lang.IllegalStateException: > >> > >> Grok parser Error: Grok statement produced a null message. Original > >> message > >> was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS > Scheduler. > >> and the parsed message was: {} > >> > >> 6. For the enrichment topology, getting the below exception. > >> 2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The > configuration > >> request.required.acks = 1 was supplied but isn't a known config. > >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version : > >> 0.10.0.1 > >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId : > >> a7a17cdec9eaa6c5 > >> 2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt > >> enrichmentErrorOutputBolt:(1) > >> 2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using > >> System classloader > >> 2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt > >> threatIntelSplitBolt:(14) > >> 2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt > >> enrichmentSplitBolt:(3) > >> 2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt > >> hostEnrichmentBolt:(5) > >> 2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt > >> enrichmentJoinBolt:(2) > >> 2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR] > >> org.apache.metron.jackson.core.JsonParseException: Unrecognized token > >> 'enrichments': was expecting ('true', 'false' or 'null') > >> at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23] > >> at > >> org.apache.metron.jackson.core.JsonParser._constructError(JsonParser. > java:1586) > >> > >> ~[stormjar.jar:?] > >> at > >> org.apache.metron.jackson.core.base.ParserMinimalBase._ > >> reportError(ParserMinimalBase.java:521) > >> ~[stormjar.jar:?] > >> at > >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._ > >> reportInvalidToken(UTF8StreamJsonParser.java:3466) > >> ~[stormjar.jar:?] > >> at > >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._ > >> handleUnexpectedValue(UTF8StreamJsonParser.java:2625) > >> ~[stormjar.jar:?] > >> at > >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._ > >> nextTokenNotInObject(UTF8StreamJsonParser.java:854) > >> ~[stormjar.jar:?] > >> at > >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken( > UTF8StreamJsonParser.java:748) > >> > >> ~[stormjar.jar:?] > >> at > >> org.apache.metron.jackson.databind.ObjectMapper._ > >> initForReading(ObjectMapper.java:3847) > >> ~[stormjar.jar:?] > >> at > >> org.apache.metron.jackson.databind.ObjectMapper._ > >> readMapAndClose(ObjectMapper.java:3792) > >> ~[stormjar.jar:?] > >> at > >> org.apache.metron.jackson.databind.ObjectMapper. > >> readValue(ObjectMapper.java:2867) > >> ~[stormjar.jar:?] > >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55) > >> ~[stormjar.jar:?] > >> > >> 7. For the Indexing topology , getting below exception > >> com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException: > >> Unrecognized token 'indexing': was expecting ('true', 'false' or 'null') > >> at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17] > >> at > >> com.fasterxml.jackson.core.metron.elasticsearch. > >> JsonParser._constructError(JsonParser.java:1581) > >> ~[stormjar.jar:?] > >> at > >> com.fasterxml.jackson.core.metron.elasticsearch.base. > ParserMinimalBase._ > >> reportError(ParserMinimalBase.java:533) > >> ~[stormjar.jar:?] > >> at > >> com.fasterxml.jackson.core.metron.elasticsearch.json. > >> UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java: > 3451) > >> ~[stormjar.jar:?] > >> at > >> com.fasterxml.jackson.core.metron.elasticsearch.json. > >> UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java: > 2610) > >> > >> ~[stormjar.jar:?] > >> at > >> com.fasterxml.jackson.core.metron.elasticsearch.json. > >> UTF8StreamJsonParser._nextTokenNotInObject( > UTF8StreamJsonParser.java:841) > >> ~[stormjar.jar:?] > >> at > >> com.fasterxml.jackson.core.metron.elasticsearch.json. > UTF8StreamJsonParser. > >> nextToken(UTF8StreamJsonParser.java:737) > >> ~[stormjar.jar:?] > >> at > >> com.fasterxml.jackson.databind.ObjectMapper._ > initForReading(ObjectMapper.java:3847) > >> > >> ~[stormjar.jar:?] > >> at > >> com.fasterxml.jackson.databind.ObjectMapper._ > readMapAndClose(ObjectMapper.java:3792) > >> > >> ~[stormjar.jar:?] > >> at > >> com.fasterxml.jackson.databind.ObjectMapper. > readValue(ObjectMapper.java:2874) > >> > >> ~[stormjar.jar:?] > >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41) > >> ~[stormjar.jar:?] > >> at > >> org.apache.metron.common.configuration.IndexingConfigurations. > >> updateSensorIndexingConfig(IndexingConfigurations.java:52) > >> ~[stormjar.jar:?] > >> > >> > >> Kindly help to resolve the issue. > >> > >> Regards > >> Girish N > >> > >> >
