[ https://issues.apache.org/jira/browse/METRON-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688683#comment-16688683 ]
ASF GitHub Bot commented on METRON-1874: ---------------------------------------- GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1265 METRON-1874 Create a Parser Debugger Users should be able to parse messages in a controlled environment like the Stellar REPL. This can help troubleshoot issues that are occurring at high velocity in a live Parser topology. * This could help a user understand why a deployed parser is not working as they would expect. * This can help a user to test their parser configuration before deploying to their live Metron cluster. * This can help a user understand why a particular message is failing to parse successfully. ## Try It Out ### Parse a Message 1. Define the parser configuration. ``` [Stellar]>>> config := SHELL_EDIT() { "parserClassName":"org.apache.metron.parsers.bro.BasicBroParser", "filterClassName":"org.apache.metron.parsers.filters.StellarFilter", "sensorTopic":"bro" } ``` 1. Grab a message from the input topic to parse. You could also just mock-up a message that you would like to test. ``` [Stellar]>>> bro := KAFKA_GET('bro') [{"http": {"ts":1542313125.807068,"uid":"CUrRne3iLIxXavQtci","id.orig_h"... ``` 1. Initialize the parser. The parser keeps track of the number of successes and failures which can be useful when parsing a batch of messages. ``` [Stellar]>>> parser := PARSER_INIT("bro", config) Parser{0 successful, 0 error(s)} ``` 1. Parse the message. ``` [Stellar]>>> msgs := PARSER_PARSE(parser, bro) [{"bro_timestamp":"1542313125.807068","method":"GET","ip_dst_port":8080,... ``` 1. Review the successfully parsed message. ``` [Stellar]>>> LENGTH(msgs) 1 ``` ``` [Stellar]>>> msg := GET(msgs, 0) [Stellar]>>> MAP_GET("guid", msg) 7f2e0c77-c58c-488e-b1ad-fbec10fb8182 ``` ``` [Stellar]>>> MAP_GET("timestamp", msg) 1542313125807 ``` ``` [Stellar]>>> MAP_GET("source.type", msg) bro ``` 1. The parser will tally the success. ``` [Stellar]>>> parser Parser{1 successful, 0 error(s)} ``` ### Parse Multiple Messages 1. Grab 5 raw input messages from Kafka. ``` [Stellar]>>> input := KAFKA_GET("bro", 5) [{"dns": {"ts":1542313125.342913,"uid":"CmJWpN3Ynwsggof57e", ... ``` 1. Parse the messages. ``` [Stellar]>>> msgs := PARSER_PARSE(parser, input) [{"TTLs":[13888.0],"qclass_name":"C_INTERNET", ... ``` 1. Review the parsed messages. There were 5 messages returned and each have a valid GUID as you would expect. ``` [Stellar]>>> LENGTH(msgs) 5 ``` ``` [Stellar]>>> MAP(msgs, m -> MAP_GET("guid", m)) [3b40ab62-ab6a-4dff-86c5-f35cdb2b01ea, 3b5826a7-f2d4-4df2-a28a-ab1f66037b4b, 9fc5f794-26f6-464f-bb99-05fb649ea465, c7162bee-01f9-4cc2-8e26-13101bc22ac1, b86dbb50-cb1d-4889-87ee-3919bcce6fdb] ```` ### Parse an Invalid Message 1. Mock-up a message that will fail to parse. ``` [Stellar]>>> invalid := "{invalid>" {invalid> ``` 1. Parse the invalid message. This will show you the actual exception that occurred along with return the error message that is pushed onto the error topic. ``` [Stellar]>>> errors := PARSER_PARSE(parser, invalid) 2018-11-15 20:29:01 ERROR BasicBroParser:144 - Unable to parse Message: {invalid> Unexpected character (i) at position 1. at org.json.simple.parser.Yylex.yylex(Yylex.java:610) at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269) at org.json.simple.parser.JSONParser.parse(JSONParser.java:118) at org.json.simple.parser.JSONParser.parse(JSONParser.java:81) at org.json.simple.parser.JSONParser.parse(JSONParser.java:75) at org.apache.metron.parsers.bro.JSONCleaner.clean(JSONCleaner.java:49) at org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:68) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:54) at org.apache.metron.parsers.interfaces.MessageParser.parseOptionalResult(MessageParser.java:67) at org.apache.metron.parsers.ParserRunnerImpl.execute(ParserRunnerImpl.java:146) at org.apache.metron.management.StellarParserRunner.lambda$doParse$3(StellarParserRunner.java:76) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at org.apache.metron.management.StellarParserRunner.doParse(StellarParserRunner.java:77) at org.apache.metron.management.StellarParserRunner.parse(StellarParserRunner.java:62) at org.apache.metron.management.ParserFunctions$ParseFunction.apply(ParserFunctions.java:98) at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652) at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250) at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151) at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:405) at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257) at org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:66) at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:252) at org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:359) at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [{"exception":"java.lang.IllegalStateException: Unable to parse Message: {invalid>","failed_sensor_type":"bro","stack":"java.lang.IllegalStateException: Unable to parse Message: {invalid>\n\tat org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:145)\n\tat org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:54)\n\tat org.apache.metron.parsers.interfaces.MessageParser.parseOptionalResult(MessageParser.java:67)\n\tat org.apache.metron.parsers.ParserRunnerImpl.execute(ParserRunnerImpl.java:146)\n\tat org.apache.metron.management.StellarParserRunner.lambda$doParse$3(StellarParserRunner.java:76)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)\n\tat java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)\n\tat java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)\n\tat java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)\n\tat java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)\n\tat org.apache.metron.management.StellarParserRunner.doParse(StellarParserRunner.java:77)\n\tat org.apache.metron.management.StellarParserRunner.parse(StellarParserRunner.java:62)\n\tat org.apache.metron.management.ParserFunctions$ParseFunction.apply(ParserFunctions.java:98)\n\tat org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652)\n\tat org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250)\n\tat org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)\n\tat org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:405)\n\tat org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257)\n\tat org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:66)\n\tat org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:252)\n\tat org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:359)\n\tat org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: Unexpected character (i) at position 1.\n\tat org.json.simple.parser.Yylex.yylex(Yylex.java:610)\n\tat org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)\n\tat org.json.simple.parser.JSONParser.parse(JSONParser.java:118)\n\tat org.json.simple.parser.JSONParser.parse(JSONParser.java:81)\n\tat org.json.simple.parser.JSONParser.parse(JSONParser.java:75)\n\tat org.apache.metron.parsers.bro.JSONCleaner.clean(JSONCleaner.java:49)\n\tat org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:68)\n\t... 28 more\n","hostname":"node1","raw_message":"{invalid>","error_hash":"e547a79488545c912977781a8d556341b3263943fad484f4d4b87e3b6052eac2","error_type":"parser_error","guid":"a2e06d23-6cef-4f26-b0ba-52263764d4ef","message":"Unable to parse Message: {invalid>","source.type":"error","timestamp":1542313741271}] ``` 1. Review the details of the error. ``` [Stellar]>>> error := GET(errors, 0) [Stellar]>>> MAP_GET("raw_message", error) {invalid> [Stellar]>>> MAP_GET("message", error) Unable to parse Message: {invalid> [Stellar]>>> MAP_GET("stack", error) java.lang.IllegalStateException: Unable to parse Message: {invalid> at org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:145) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:54) at org.apache.metron.parsers.interfaces.MessageParser.parseOptionalResult(MessageParser.java:67) at org.apache.metron.parsers.ParserRunnerImpl.execute(ParserRunnerImpl.java:146) at org.apache.metron.management.StellarParserRunner.lambda$doParse$3(StellarParserRunner.java:76) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at org.apache.metron.management.StellarParserRunner.doParse(StellarParserRunner.java:77) at org.apache.metron.management.StellarParserRunner.parse(StellarParserRunner.java:62) at org.apache.metron.management.ParserFunctions$ParseFunction.apply(ParserFunctions.java:98) at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652) at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250) at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151) at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:405) at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257) at org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:66) at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:252) at org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:359) at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: Unexpected character (i) at position 1. at org.json.simple.parser.Yylex.yylex(Yylex.java:610) at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269) at org.json.simple.parser.JSONParser.parse(JSONParser.java:118) at org.json.simple.parser.JSONParser.parse(JSONParser.java:81) at org.json.simple.parser.JSONParser.parse(JSONParser.java:75) at org.apache.metron.parsers.bro.JSONCleaner.clean(JSONCleaner.java:49) at org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:68) ... 28 more ``` ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1874 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1265.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1265 ---- commit 98155b884a8039017a244c18bd20d13989ab69f7 Author: Nick Allen <nick@...> Date: 2018-10-18T23:38:47Z Initial pass commit 9aa0c366fdff0ef40c7fdb389e4f5dc0316d637b Author: Nick Allen <nick@...> Date: 2018-11-15T19:10:49Z Able to parse one or messages in the REPL commit 7854a2f75c47b72c51150902c03458076c00d87b Author: Nick Allen <nick@...> Date: 2018-11-15T21:28:22Z Tallying successes and errors. Added config function. ---- > Create a Parser Debugger > ------------------------ > > Key: METRON-1874 > URL: https://issues.apache.org/jira/browse/METRON-1874 > Project: Metron > Issue Type: Bug > Reporter: Nick Allen > Assignee: Nick Allen > Priority: Major > > Users need a way to test their parser configurations in a controlled > environment. This would allow a user to tweak and test the parser > configuration before deploying it into a live Metron cluster. The user could > also use the REPL to understand why a particular message failed to parse > correctly. > This could follow the pattern of the [Profiler > Debugger|https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles] > and [Threat Triage > Simulator|https://github.com/apache/metron/tree/master/metron-platform/metron-management#simulate-threat-triage-rules]. > This allows a user to apply canned messages against a configuration and > inspect the output, all within the REPL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)