GitHub user ottobackwards opened a pull request:
https://github.com/apache/metron/pull/914
METRON-1397 Support for JSON Path and complex documents in JSONMapParser
It would be useful for implementors to have the ability to ingest more
complex documents without having to stand up a NiFi node/cluster.
This PR adds support for splitting multiple messages from a single message
by way of using JSON Path statements to select the List element in the document.
Since the parse() interface already returns a List<JSONObject> the scope of
this change is in the parser.
Example:
```json
{
"foo" :
[
{ "name" : "foo1", "value" : "bar", "number" : 1.0 },
{ "name" : "foo2", "value" : "baz", "number" : 2.0 }
]
}
```
with JSON Path statement "$.foo" will result in two messages emitted from
the parser
```json
{ "name" : "foo1", "value" : "bar", "number" : 1.0 }
```
```json
{ "name" : "foo2", "value" : "baz", "number" : 2.0 }
```
Support in the configuration for a new configuration parameter: jsonpQuery
has been added.
An example of the new configuration:
```json
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"jsonMapQuery",
"parserConfig": {"jsonpQuery":"$.foo"}
}
```
In order to integration test both with and without the query present, a new
jsonMapQuery parser configuration has been added, with test data.
##Testing
- tests should run
- deployment should work
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/metron jsonp-support
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/914.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #914
----
commit 414611b9ea6eb06b79cba0cc86e63ea062335884
Author: Otto Fowler <ottobackwards@...>
Date: 2018-01-29T16:58:16Z
JSON Path support for JSONMapParser
Tests, Integration Tests, Doc
commit ad03ac4d3e7c98fcf571f3f3906efbb915deb1f9
Author: Otto Fowler <ottobackwards@...>
Date: 2018-01-29T17:13:10Z
checkstyle fixes
commit 7c76c5f5b7507ac55861ed44ad131700c1e6f124
Author: Otto Fowler <ottobackwards@...>
Date: 2018-01-29T19:36:16Z
account for new config in rpm
fix handle empty
----
---
