GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/1084
METRON-1644: Support parser chaining
## Contributor Comments
Currently we have only one layer of parsing prior to enrichment, but often
real data is more complex. For instance, often data is wrapped in an envelope
(e.g. syslog data which contains a field which needs to be parsed).
This Jira allows us to support a DAG of parsers prior to enrichment by
allowing us to provide a strategy for interpreting what is data and what is
metadata in the parser bolt. This enables upstream parsers to pass in a JSON
Blob which contains the metadata and have the parser bolt choose which field is
the data to be parsed, the remaining fields would be considered metadata (and
merged into the resulting data or not depending on our existing rules for
handling metadata).
To illustrate this better, I've provided a use-case with an example. Note,
this PR depends on METRON-1643 and METRON-1642, so those should be reviewed
prior to this.
Manual Test script:
* Go through the use-case doc.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron
enveloped_message_supplier
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1084.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1084
----
commit 5030d00f9eb13ff662716f1c05a731938a66cb52
Author: cstella <cestella@...>
Date: 2018-06-05T19:05:59Z
Regex based message router transformer.
commit d1cb318961277da732650152c8526054769d830b
Author: cstella <cestella@...>
Date: 2018-06-05T21:07:13Z
Adding message supplier. Tests pending.
commit ffad876b377f6cf94213b783a8c139e0973c2e00
Author: cstella <cestella@...>
Date: 2018-06-06T14:12:55Z
Fixed tests
commit b349bf0b8fa251846d77cabc8b6aa019154a6425
Author: cstella <cestella@...>
Date: 2018-06-06T14:22:53Z
missing license
commit 7c2723bc243996a2a0606e07def5e3b5c1736e59
Author: cstella <cestella@...>
Date: 2018-06-06T14:35:25Z
updating test
commit 71959c89816ddf4d84fef9585cfb03bd7f624612
Author: cstella <cestella@...>
Date: 2018-06-07T16:05:14Z
Renamed and refactored metadata utilities
commit 6e809e4cf9c34c9207d2d7c501db1dc0390c1d0b
Author: cstella <cestella@...>
Date: 2018-06-07T16:06:51Z
compilation issues
commit 5c9dd7caa68820d1ca06d99ff3ef2ac7eace7ba6
Author: cstella <cestella@...>
Date: 2018-06-07T20:55:10Z
Added unit and integration tests
commit bbd922e440e3dcfd22d54e76710d75d3fd2f5ab9
Author: cstella <cestella@...>
Date: 2018-06-07T22:10:00Z
Updating documentation and refactoring.
commit b2d1ca8b685306aafe07a65811f8223b6136bf9a
Author: cstella <cestella@...>
Date: 2018-06-08T02:36:49Z
Updated integration test.
commit 4a5461a8a7b4cd34d6541c0449470901d6ef3ed5
Author: cstella <cestella@...>
Date: 2018-06-08T02:56:50Z
Merge branch 'master' into regex_routing
commit f1bda7478824e716d60a28cab8680e303a9e28a7
Author: cstella <cestella@...>
Date: 2018-06-08T03:03:34Z
regex routing docs
commit 3b01987a64030a1dd3cccac2d70d7e5d375bac1c
Author: cstella <cestella@...>
Date: 2018-06-08T03:07:04Z
Merge branch 'master' into enveloped_message_supplier
commit f940234aec3e3360e12b4015c9cdc9cf01df681d
Author: cstella <cestella@...>
Date: 2018-06-08T03:30:07Z
Fixed test and subtle bug.
commit 92e92fa658d5aaeeb92b970449ac7dded0ed675e
Author: cstella <cestella@...>
Date: 2018-06-08T05:06:53Z
typos
commit 8c7112095ceddf5b7a1e812b905721991abd91ee
Author: cstella <cestella@...>
Date: 2018-06-08T05:11:02Z
typo
commit b9036f8f6feebbb19efb964a66f5ff8131fcbe8d
Author: cstella <cestella@...>
Date: 2018-06-08T05:21:28Z
missed sensorname
commit 532add3d533bd73ab7228a09e0dfc50255242c33
Author: cstella <cestella@...>
Date: 2018-06-08T06:21:44Z
updating grok
commit b5140a22fda7fd39f18b5a13c91a95785f3b6a4b
Author: cstella <cestella@...>
Date: 2018-06-08T06:33:38Z
Updating grok
commit ba862789e43f9e5d8e648d86378d89ec98d30d5b
Author: cstella <cestella@...>
Date: 2018-06-27T14:19:42Z
Merge branch 'master' into regex_routing
commit cfcbcff3353a6773c18cebfa877e350a85e38859
Author: cstella <cestella@...>
Date: 2018-06-27T14:19:55Z
Merge branch 'master' into enveloped_message_supplier
commit e807b0d7b61ac8b77b24051c8e0eedb53aa9be9d
Author: cstella <cestella@...>
Date: 2018-06-27T15:11:25Z
Phrasing! we ARE still doing phrasing, right?
commit 8a96773c72077785eff538abbf359587b71d8e28
Author: cstella <cestella@...>
Date: 2018-06-27T15:22:39Z
Renaming to REGEX_SELECT
commit 28cbfaff4a24de43c4752761dfbfa9e7d5577274
Author: cstella <cestella@...>
Date: 2018-06-27T15:22:53Z
Merge branch 'regex_routing' into enveloped_message_supplier
commit 9218b469367a917b0fb004a473f06188483a10be
Author: cstella <cestella@...>
Date: 2018-06-27T15:23:29Z
Updating naming change
commit 2cd91268c0a8de9d86b01e5613828f6ded8deb9b
Author: cstella <cestella@...>
Date: 2018-06-27T21:26:51Z
Creating a parser chaining document
commit c4e4786e778d5b06cd16f7faa7d3522f620fc2ba
Author: cstella <cestella@...>
Date: 2018-07-06T15:44:41Z
Merge branch 'master' into enveloped_message_supplier
commit 2623660f9191a7596c53b623c031e13b75e92fd7
Author: cstella <cestella@...>
Date: 2018-07-06T18:35:35Z
Updating with javadocs
commit efbe06e245506a9a25c0fd1cf5a3d173fb8f1f81
Author: cstella <cestella@...>
Date: 2018-07-06T18:50:53Z
Fixed readme
----
---
