GitHub user merrimanr opened a pull request:
https://github.com/apache/incubator-metron/pull/120
Metron 142 Simplify Parser configuration
This pull request includes several changes related to optimizing
configuration for parser topologies. Currently the process for adding parsers
includes several unnecessary steps with configuration being duplicated in
various places. The high level changes are:
- Sensor specific config files (parser and enrichment) were reorganized
into their respective projects
- Parser topology configuration is now dynamically driven by simple sensor
specific json config files
- Zookeeper configuration infrastructure was extended to support parser
topologies
- Ansible scripts were updated to load parser configurations to zookeeper
and call parser topology startup scripts correctly.
- Unit and integration tests were updated to accommodate parser topology
changes and provide coverage for new classes.
- Updated tools for reading and writing sensor parser configs to zookeeper
- Removed generic configuration feature as it is not being used anywhere
- Changed enrichment integration and unit tests to use a dedicated test
config instead of reusing a sensor specific enrichment config
- Moved zookeeper configuration functions to a new component in the
integration testing framework.
- Abstracted integration test validations to facilitate more readable code
and reusable validations
The major benefits of this pull request are a simpler process for adding
more parsers and additional test coverage for sensor specific parser
configurations. This has been integration tested and verified with a
full-dev-platform build.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-142
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/120.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #120
----
commit a8b2ece52d60ba7c69685455b38d3e71ec052979
Author: rmerriman <[email protected]>
Date: 2016-05-02T14:39:33Z
moved parser topology from flux to java
commit d8317b14176278be3ef1bba4b8bb5b5423b83b57
Author: rmerriman <[email protected]>
Date: 2016-05-03T22:31:58Z
finished integration test
commit 3b20cf7879faae9791856da08dc4dde25bad1a71
Author: rmerriman <[email protected]>
Date: 2016-05-05T15:51:13Z
Merge remote-tracking branch 'apache/master' into parser-refactor
Conflicts:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/GrokParser.java
commit 136f76a539fc2378a63086f36c09271c950ccc3d
Author: rmerriman <[email protected]>
Date: 2016-05-06T18:19:58Z
Merge remote-tracking branch 'apache/master' into parser-refactor
commit 7e511d49c50f588207bf16af26450e33b76c5152
Author: rmerriman <[email protected]>
Date: 2016-05-06T22:36:25Z
METRON-142: Simplify Parser configuration
commit 5a05dcbed7bb235fbc0feedd05b0da26d7ce6ca6
Author: rmerriman <[email protected]>
Date: 2016-05-06T22:43:47Z
METRON-142: Simplify Parser configuration
commit 5bebc79fea5344fe3799431a52114b4c4672c61b
Author: rmerriman <[email protected]>
Date: 2016-05-06T23:25:52Z
Removed old parser integration tests
commit 79d02def004f12f6df4387ad9b5a26963bc46916
Author: rmerriman <[email protected]>
Date: 2016-05-07T17:11:58Z
added mvn clean to travis build
commit de4292c6d1fed4d86ad1c29c24d322edf7b562cb
Author: rmerriman <[email protected]>
Date: 2016-05-09T13:43:47Z
Removed uneeded failing pcap service test
commit 109b543f3cc04cd5a4066f85af9c47e874eb36a6
Author: rmerriman <[email protected]>
Date: 2016-05-11T13:18:45Z
Merge remote-tracking branch 'apache/master' into METRON-142
Conflicts:
metron-deployment/roles/metron_streaming/files/config/sensors/pcap.json
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
metron-platform/metron-common/src/test/java/org/apache/metron/common/bolt/ConfiguredEnrichmentBoltTest.java
metron-platform/metron-common/src/test/java/org/apache/metron/common/cli/ConfigurationsUtilsTest.java
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriter.java
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/BulkMessageWriterBolt.java
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/utils/SampleUtil.java
commit c8a142ec670316580690101705e136d914942169
Author: rmerriman <[email protected]>
Date: 2016-05-12T15:43:53Z
Resolve merge conflicts
commit e33119529c0c2b4843ba9f3c77f13bd43a93b370
Author: rmerriman <[email protected]>
Date: 2016-05-12T16:38:59Z
Sensor Parser integration tests now implemented in separate classes
commit 44494198827ec1d92dfeabe8697ceffe22b1e773
Author: rmerriman <[email protected]>
Date: 2016-05-12T20:40:06Z
Merge remote-tracking branch 'apache/master' into METRON-142
commit 512b7e7ac9db0238a7a22bd1c4a42de534a969e7
Author: rmerriman <[email protected]>
Date: 2016-05-12T20:46:22Z
Removed old pcap.json file
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
