GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1265
METRON-1874 Create a Parser Debugger
Users should be able to parse messages in a controlled environment like the
Stellar REPL. This can help troubleshoot issues that are occurring at high
velocity in a live Parser topology.
* This could help a user understand why a deployed parser is not working as
they would expect.
* This can help a user to test their parser configuration before deploying
to their live Metron cluster.
* This can help a user understand why a particular message is failing to
parse successfully.
## Try It Out
### Parse a Message
1. Define the parser configuration.
```
[Stellar]>>> config := SHELL_EDIT()
{
"parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
"filterClassName":"org.apache.metron.parsers.filters.StellarFilter",
"sensorTopic":"bro"
}
```
1. Grab a message from the input topic to parse. You could also just
mock-up a message that you would like to test.
```
[Stellar]>>> bro := KAFKA_GET('bro')
[{"http":
{"ts":1542313125.807068,"uid":"CUrRne3iLIxXavQtci","id.orig_h"...
```
1. Initialize the parser. The parser keeps track of the number of successes
and failures which can be useful when parsing a batch of messages.
```
[Stellar]>>> parser := PARSER_INIT("bro", config)
Parser{0 successful, 0 error(s)}
```
1. Parse the message.
```
[Stellar]>>> msgs := PARSER_PARSE(parser, bro)
[{"bro_timestamp":"1542313125.807068","method":"GET","ip_dst_port":8080,...
```
1. Review the successfully parsed message.
```
[Stellar]>>> LENGTH(msgs)
1
```
```
[Stellar]>>> msg := GET(msgs, 0)
[Stellar]>>> MAP_GET("guid", msg)
7f2e0c77-c58c-488e-b1ad-fbec10fb8182
```
```
[Stellar]>>> MAP_GET("timestamp", msg)
1542313125807
```
```
[Stellar]>>> MAP_GET("source.type", msg)
bro
```
1. The parser will tally the success.
```
[Stellar]>>> parser
Parser{1 successful, 0 error(s)}
```
### Parse Multiple Messages
1. Grab 5 raw input messages from Kafka.
```
[Stellar]>>> input := KAFKA_GET("bro", 5)
[{"dns": {"ts":1542313125.342913,"uid":"CmJWpN3Ynwsggof57e", ...
```
1. Parse the messages.
```
[Stellar]>>> msgs := PARSER_PARSE(parser, input)
[{"TTLs":[13888.0],"qclass_name":"C_INTERNET", ...
```
1. Review the parsed messages. There were 5 messages returned and each
have a valid GUID as you would expect.
```
[Stellar]>>> LENGTH(msgs)
5
```
```
[Stellar]>>> MAP(msgs, m -> MAP_GET("guid", m))
[3b40ab62-ab6a-4dff-86c5-f35cdb2b01ea,
3b5826a7-f2d4-4df2-a28a-ab1f66037b4b, 9fc5f794-26f6-464f-bb99-05fb649ea465,
c7162bee-01f9-4cc2-8e26-13101bc22ac1, b86dbb50-cb1d-4889-87ee-3919bcce6fdb]
````
### Parse an Invalid Message
1. Mock-up a message that will fail to parse.
```
[Stellar]>>> invalid := "{invalid>"
{invalid>
```
1. Parse the invalid message. This will show you the actual exception that
occurred along with return the error message that is pushed onto the error
topic.
```
[Stellar]>>> errors := PARSER_PARSE(parser, invalid)
2018-11-15 20:29:01 ERROR BasicBroParser:144 - Unable to parse Message:
{invalid>
Unexpected character (i) at position 1.
at org.json.simple.parser.Yylex.yylex(Yylex.java:610)
at
org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
at
org.apache.metron.parsers.bro.JSONCleaner.clean(JSONCleaner.java:49)
at
org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:68)
at
org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:54)
at
org.apache.metron.parsers.interfaces.MessageParser.parseOptionalResult(MessageParser.java:67)
at
org.apache.metron.parsers.ParserRunnerImpl.execute(ParserRunnerImpl.java:146)
at
org.apache.metron.management.StellarParserRunner.lambda$doParse$3(StellarParserRunner.java:76)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at
org.apache.metron.management.StellarParserRunner.doParse(StellarParserRunner.java:77)
at
org.apache.metron.management.StellarParserRunner.parse(StellarParserRunner.java:62)
at
org.apache.metron.management.ParserFunctions$ParseFunction.apply(ParserFunctions.java:98)
at
org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652)
at
org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250)
at
org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
at
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:405)
at
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257)
at
org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:66)
at
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:252)
at
org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:359)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[{"exception":"java.lang.IllegalStateException: Unable to parse
Message:
{invalid>","failed_sensor_type":"bro","stack":"java.lang.IllegalStateException:
Unable to parse Message: {invalid>\n\tat
org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:145)\n\tat
org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:54)\n\tat
org.apache.metron.parsers.interfaces.MessageParser.parseOptionalResult(MessageParser.java:67)\n\tat
org.apache.metron.parsers.ParserRunnerImpl.execute(ParserRunnerImpl.java:146)\n\tat
org.apache.metron.management.StellarParserRunner.lambda$doParse$3(StellarParserRunner.java:76)\n\tat
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)\n\tat
java.util.
stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)\n\tat
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)\n\tat
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)\n\tat
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)\n\tat
org.apache.metron.management.StellarParserRunner.doParse(StellarParserRunner.java:77)\n\tat
org.apache.metron.management.StellarParserRunner.parse(StellarParserRunner.java:62)\n\tat
org.apache.metron.management.ParserFunctions$ParseFunction.apply(ParserFunctions.java:98)\n\tat
org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652)\n\tat
org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250)\n\tat
org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)\n\tat
org.apache.metron.stellar.c
ommon.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:405)\n\tat
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257)\n\tat
org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:66)\n\tat
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:252)\n\tat
org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:359)\n\tat
org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
java.lang.Thread.run(Thread.java:748)\nCaused by: Unexpected character (i) at
position 1.\n\tat org.json.simple.parser.Yylex.yylex(Yylex.java:610)\n\tat
org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)\n\ta
t org.json.simple.parser.JSONParser.parse(JSONParser.java:118)\n\tat
org.json.simple.parser.JSONParser.parse(JSONParser.java:81)\n\tat
org.json.simple.parser.JSONParser.parse(JSONParser.java:75)\n\tat
org.apache.metron.parsers.bro.JSONCleaner.clean(JSONCleaner.java:49)\n\tat
org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:68)\n\t...
28
more\n","hostname":"node1","raw_message":"{invalid>","error_hash":"e547a79488545c912977781a8d556341b3263943fad484f4d4b87e3b6052eac2","error_type":"parser_error","guid":"a2e06d23-6cef-4f26-b0ba-52263764d4ef","message":"Unable
to parse Message: {invalid>","source.type":"error","timestamp":1542313741271}]
```
1. Review the details of the error.
```
[Stellar]>>> error := GET(errors, 0)
[Stellar]>>> MAP_GET("raw_message", error)
{invalid>
[Stellar]>>> MAP_GET("message", error)
Unable to parse Message: {invalid>
[Stellar]>>> MAP_GET("stack", error)
java.lang.IllegalStateException: Unable to parse Message: {invalid>
at
org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:145)
at
org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:54)
at
org.apache.metron.parsers.interfaces.MessageParser.parseOptionalResult(MessageParser.java:67)
at
org.apache.metron.parsers.ParserRunnerImpl.execute(ParserRunnerImpl.java:146)
at
org.apache.metron.management.StellarParserRunner.lambda$doParse$3(StellarParserRunner.java:76)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at
org.apache.metron.management.StellarParserRunner.doParse(StellarParserRunner.java:77)
at
org.apache.metron.management.StellarParserRunner.parse(StellarParserRunner.java:62)
at
org.apache.metron.management.ParserFunctions$ParseFunction.apply(ParserFunctions.java:98)
at
org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652)
at
org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250)
at
org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
at
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:405)
at
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257)
at
org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:66)
at
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:252)
at
org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:359)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: Unexpected character (i) at position 1.
at org.json.simple.parser.Yylex.yylex(Yylex.java:610)
at
org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
at
org.apache.metron.parsers.bro.JSONCleaner.clean(JSONCleaner.java:49)
at
org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:68)
... 28 more
```
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1874
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1265.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1265
----
commit 98155b884a8039017a244c18bd20d13989ab69f7
Author: Nick Allen <nick@...>
Date: 2018-10-18T23:38:47Z
Initial pass
commit 9aa0c366fdff0ef40c7fdb389e4f5dc0316d637b
Author: Nick Allen <nick@...>
Date: 2018-11-15T19:10:49Z
Able to parse one or messages in the REPL
commit 7854a2f75c47b72c51150902c03458076c00d87b
Author: Nick Allen <nick@...>
Date: 2018-11-15T21:28:22Z
Tallying successes and errors. Added config function.
----
---
