[
https://issues.apache.org/jira/browse/METRON-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16714178#comment-16714178
]
ASF GitHub Bot commented on METRON-1892:
----------------------------------------
GitHub user nickwallen reopened a pull request:
https://github.com/apache/metron/pull/1278
METRON-1892 Parser Debugger Should Load Config From Zookeeper
When using the parser debugger functions created in #1265, the user has to
manually specify the parser configuration. This is useful when testing out a
new parser before it goes live.
In other cases, it would be simpler to use the sensor configuration values
that are 'live' and already loaded in Zookeeper. For example, I might want to
test why a particular messages fails to parse in my environment.
## Try It Out
Try out the following examples in the Stellar REPL.
1. Launch a development environment.
1. Launch the REPL.
```
source /etc/default/metron
cd $METRON_HOME
bin/stellar -z $ZOOKEEPER
```
### Parse a Message
1. Grab a message from the input topic to parse. You could also just
mock-up a message that you would like to test.
```
[Stellar]>>> input := KAFKA_GET('bro')
[{"http":
{"ts":1542313125.807068,"uid":"CUrRne3iLIxXavQtci","id.orig_h"...
```
1. Initialize the parser. The parser configuration for 'bro' will be loaded
automatically from Zookeeper.
```
[Stellar]>>> parser := PARSER_INIT("bro")
Parser{0 successful, 0 error(s)}
```
1. Parse the message.
```
[Stellar]>>> msgs := PARSER_PARSE(parser, input)
[{"bro_timestamp":"1542313125.807068","method":"GET","ip_dst_port":8080,...
```
1. The parser will tally the success.
```
[Stellar]>>> parser
Parser{1 successful, 0 error(s)}
```
1. Review the successfully parsed message.
```
[Stellar]>>> LENGTH(msgs)
1
```
```
[Stellar]>>> msg := GET(msgs, 0)
[Stellar]>>> MAP_GET("guid", msg)
7f2e0c77-c58c-488e-b1ad-fbec10fb8182
```
```
[Stellar]>>> MAP_GET("timestamp", msg)
1542313125807
```
```
[Stellar]>>> MAP_GET("source.type", msg)
bro
```
### Missing Configuration
1. If the configuration does not exist in Zookeeper, you should see
something like this.
I have not configured a parser named 'tuna' in my environment (but I
could go for a tuna sandwich right about now ).
```
[Stellar]>>> bad := PARSER_INIT('tuna')
[!] Unable to parse: PARSER_INIT('tuna') due to: Unable to read
configuration from Zookeeper; sensorType = tuna
org.apache.metron.stellar.dsl.ParseException: Unable to parse:
PARSER_INIT('tuna') due to: Unable to read configuration from Zookeeper;
sensorType = tuna
at
org.apache.metron.stellar.common.BaseStellarProcessor.createException(BaseStellarProcessor.java:166)
at
org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:154)
at
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:405)
at
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257)
at
org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:66)
at
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:252)
at
org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:359)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Unable to read
configuration from Zookeeper; sensorType = tuna
at
org.apache.metron.management.ParserFunctions$InitializeFunction.readFromZookeeper(ParserFunctions.java:103)
at
org.apache.metron.management.ParserFunctions$InitializeFunction.apply(ParserFunctions.java:66)
at
org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:661)
at
org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:259)
at
org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
... 9 more
```
## Pull Request Checklist
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1892
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1278.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1278
----
commit b56e736bafe93dd6640be90ab3b144ff7f7a918a
Author: Nick Allen <nick@...>
Date: 2018-11-21T16:43:39Z
METRON-1892 Parser Debugger Should Load Config From Zookeeper
commit cc29b6d378e5b590b146ba6aba4268dd449789a8
Author: Nick Allen <nick@...>
Date: 2018-12-08T16:21:39Z
Merge remote-tracking branch 'apache/master' into METRON-1892
----
> Parser Debugger Should Load Config From Zookeeper
> -------------------------------------------------
>
> Key: METRON-1892
> URL: https://issues.apache.org/jira/browse/METRON-1892
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
> Priority: Major
>
> When using the parser debugger functions created in METRON-1874, the user has
> to manually specify the parser configuration. This is useful when testing
> out a new parser before it goes live.
> In other cases, it would be simpler to use the configuration values that are
> 'live' and loaded in Zookeeper. For example, I might want to test why a
> particular messages fails to parse in my environment.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
