[GitHub] metron issue #531: METRON-854 create dhcp dump parser
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/531 So we are going to close this? ---
[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/687 Bump @cestella ---
[GitHub] metron issue #526: Metron-846: Add E2E tests for metron management ui
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/526 @iraghumitra what is the story with this? ---
[GitHub] metron issue #687: METRON-1563 [DISCUSS] Add Assignment to Stellar Language
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/687 I have renamed in preparation for the Feature branch based on the original jira ( create a new subtask to land this on feature and change to that ID ) ---
[GitHub] metron pull request #687: METRON-1563 [DISCUSS] Add Assignment to Stellar La...
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/687 ---
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 @cestella I rebased this on the new feature branch ( after rebasing on the same master ) and I get all of these other commits. I don't know how to get rid of them? ---
[GitHub] metron pull request #1014: METRON-1563 : Base Stellar assign for feature bra...
GitHub user ottobackwards opened a pull request:
https://github.com/apache/metron/pull/1014
METRON-1563 : Base Stellar assign for feature branch
repackage:
https://github.com/apache/metron/pull/687
Please sanity check and see that PR
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/metron stellar_assign
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1014.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1014
commit 37587689e9feadeb54465aa7319f114a4e11255b
Author: cstella <cestella@...>
Date: 2017-12-12T20:24:07Z
METRON-1306: When index template install fails, we should fail the install
closes apache/incubator-metron#834
commit 6ca08b9f598b649d60c6b6468164a374b7f6f555
Author: MohanDV <mohan.dv@...>
Date: 2017-12-13T15:55:04Z
METRON-1343 Swagger UI for User Controller needs request method (MohanDV
via ottobackwards) closes apache/metron#862
commit d9ed1bad1f5b2e2471fbea11353f2947f7f52e13
Author: nickwallen <nick@...>
Date: 2017-12-14T15:59:27Z
METRON-1349 Full Dev Builds Metron Twice (nickwallen) closes
apache/metron#866
commit c4cee6af64eda4db4da3eff86abab7d4ae4ec56a
Author: mmiklavc <michael.miklavcic@...>
Date: 2017-12-14T20:29:49Z
METRON-1345: Update EC2 README for custom Ansible (mmiklavc via mmiklavc)
closes apache/metron#859
commit d446da8e707b1069576b049452484e088a3eeede
Author: nickwallen <nick@...>
Date: 2017-12-19T17:42:39Z
METRON-1372 Validate JIRA for Releases (nickwallen) closes apache/metron#874
commit adb024070d5d098909fb3800875d0042aeb27c92
Author: ottobackwards <ottobackwards@...>
Date: 2017-12-19T20:13:22Z
METRON-1374 Script the RC checking process (ottobackwards) closes
apache/metron#876
commit 3f0b1b7b4a002d3f364bd2aee7b5921c0435c4a4
Author: cstella <cestella@...>
Date: 2017-12-20T14:30:03Z
METRON-1350: Add reservoir sampling functions to Stellar closes
apache/incubator-metron#867
commit 196da12c43337d019a52b99bf6178fbda45f886d
Author: nickwallen <nick@...>
Date: 2017-12-21T14:04:49Z
METRON-1348 Metron Service Checks Use Wrong Hostname (nickwallen) closes
apache/metron#864
commit 76bed5d754fcf358809f0be7a034758b9b20fc5e
Author: cstella <cestella@...>
Date: 2017-12-21T21:49:31Z
METRON-1365: Allow PROFILE_GET to return a default value for a profile and
entity that does not have a value written. closes apache/incubator-metron#871
commit 3612a89216bd57c40a1bc3e27853c6146b471e1e
Author: ottobackwards <ottobackwards@...>
Date: 2017-12-25T20:44:45Z
METRON-1376 RC Check Script should have named parameters (ottobackwards via
nickwallen) closes apache/metron#877
commit fc8723e461d655e315d0b51acd1a31f82b4efd1f
Author: nickwallen <nick@...>
Date: 2017-12-27T18:25:53Z
METRON-1351 Create Installable Packages for Ubuntu Trusty (nickwallen)
closes apache/metron#868
commit 0518408513ed54df8dbe234027b353bed2e61943
Author: mattf-horton <mfoley@...>
Date: 2017-12-31T22:01:29Z
METRON-1373 RAT failure for metron-interface/metron-alerts (mattf-horton)
closes apache/metron#875
commit 3b10f84cc49993a1c5917f54be6ca313c8d780c4
Author: justinleet <justinjleet@...>
Date: 2018-01-01T23:49:57Z
METRON-1071 Create CONTRIBUTING.md (justinleet) closes apache/metron#881
commit 2d9d7a5f6302267edafba772268145e76751795c
Author: justinleet <justinjleet@...>
Date: 2018-01-02T16:38:54Z
METRON-1381 Add Apache license to MD files and remove the Rat exclusion
(justinleet) closes apache/metron#883
commit 8a61b96b6b01b247a8ff8d800730378b8da23471
Author: mattf-horton <mfoley@...>
Date: 2018-01-02T17:55:52Z
METRON-1384 Increment master version number to 0.4.3 for on-going
development (mattf-horton via nickwallen) closes apache/metron#885
commit 01c26a77b1041204b0bbbc544cc0a5d02e9339a8
Author: nickwallen <nick@...>
Date: 2018-01-03T14:52:57Z
METRON-1362 Improve Metron Deployment README (nickwallen) closes
apache/metron#869
commit 3381b853dca1c08a7a083593045dec2c7d4d92db
Author: mattf-horton <mfoley@...>
Date: 2018-01-04T20:30:30Z
METRON-1388 update public web site to point at 0.4.2 new release
(mattf-horton) closes apache/metron#887
commit 9108072756b6ffeedade985d3cd52ef7338cd61a
Author: merrimanr <merrimanr@...>
Date: 2018-01-08T14:18:30Z
METRON-1385 Missing properties in index template causes
ElasticsearchColumnMetadataDao.getColumnMetadata to fail (merrimanr) closes
apache/metron#886
commit 0996b7348eca14fea1b1b3c4dd57861b3a30bdeb
Author: cstella <cestella@...>
Date: 2018-01-08T14:30:58Z
METRON-1377: Stellar function to generate typosquatted domains (similar to
dnstwist) closes apache/i
[GitHub] metron issue #754: METRON-1184 EC2 Deployment - Updating control_path to acc...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/754 @as22323 I need a real name to use in the commit ---
[GitHub] metron issue #687: METRON-1090 [DISCUSS] Add Assignment to Stellar Language
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/687 Ok, so we are on the same page. I purposely did not put the update in the map resolver because I knew it warranted more discussion etc, and this is it. I just wasn't sure I was understanding what you were saying exactly, and honestly, every time i go back to read from the start @mattf-horton 's comments make me mush minded ;) Now we have to pick one of the options above to continue with. ---
[GitHub] metron issue #754: METRON-1184 EC2 Deployment - Updating control_path to acc...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/754 +1 - i'll merge ---
[GitHub] metron issue #754: METRON-1184 EC2 Deployment - Updating control_path to acc...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/754 Thanks for the contribution! Please take care of the Jira ---
[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/687 I don't quite get what you are saying. I can't wrap my head around it. If you look at the example from the PR description under concept, that doesn't work without the resolver being updated. I have to be missing something ---
[GitHub] metron issue #1009: METRON-1549: Add empty object test to WriterBoltIntegrat...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1009 I think this is good. My work on the builder/Processor stuff was a decent attempt to reduce duplicate code in the integration tests without a total rethink. I think what we see across the integration tests is so much commonality that it is apparent that some re-think could be useful to have more standardize test classes and use cases structured such that there is less overhead of duplicated code required. As I have done some work on other projects and seen other approaches this has only become more clear. commons-vfs for example runs the same tests against every provider, with each provider providing specialization of it's suites. It would seem that we should be able to have a more generic suite or testing harness for *n* topologies/topics working together, as a next step refactoring of my prior `Process` work. I don't think this is the PR for that though ---
[GitHub] metron issue #1009: METRON-1549: Add empty object test to WriterBoltIntegrat...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1009 +1 ---
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1019 This pr is against the feature branch @JonZeolla so, it is not in play ---
[GitHub] metron issue #973: METRON-1356: Add a mechanism in Java for discovering serv...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/973 - Followed the steps ( we need a docker cheat sheet for deleting existing old machine/containers btw ) - ran the install/integration-test not sure it is related: ```bash [ERROR] Failed to execute goal org.apache.maven.plugins:maven-jar-plugin:3.0.2:jar (default-jar) on project metron-hbase-client: You have to use a classifier to attach supplemental artifacts to the project instead of replacing them. -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :metron-hbase-client ``` ---
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 ok, @cestella I have fixed up the feature branch and this pr so it is clean ---
[GitHub] metron issue #1015: METRON-1544 Flaky test: org.apache.metron.stellar.common...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1015 Is there not some relevant Caffeine test we can ape for this? ---
[GitHub] metron pull request #1021: METRON-1568: Stellar should have a _ special vari...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1021#discussion_r189861295
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
---
@@ -142,8 +143,14 @@ else if(kv.getValue() instanceof List) {
{
--- End diff --
Why would this be an either or situation?
What if I have a 'match' expression that uses both _ and other vars?
---
[GitHub] metron pull request #1021: METRON-1568: Stellar should have a _ special vari...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1021#discussion_r189862557 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/ConcatMap.java --- @@ -0,0 +1,202 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.stellar.common.utils; + + +import com.esotericsoftware.kryo.Kryo; +import com.esotericsoftware.kryo.KryoSerializable; +import com.esotericsoftware.kryo.io.Input; +import com.esotericsoftware.kryo.io.Output; +import com.google.common.base.Joiner; +import com.google.common.collect.Iterables; +import com.google.common.collect.Sets; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.Set; + --- End diff -- Can throw some java doc on this about what it is supposed to provide? And the assumptions that it makes ( like the uniqueness of the keys in the different maps and what happens etc ) ---
[GitHub] metron pull request #:
Github user ottobackwards commented on the pull request: https://github.com/apache/metron/commit/2038df3c692effafc584ef32e2eb84bed905ff3f#commitcomment-29081657 In metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/ConcatMap.java: In metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/ConcatMap.java on line 44: If a map doesn't have unique keys, is it *really* a map? ---
[GitHub] metron pull request #1021: METRON-1568: Stellar should have a _ special vari...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1021#discussion_r189898485
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
---
@@ -142,8 +143,14 @@ else if(kv.getValue() instanceof List) {
{
--- End diff --
I guess I'm wondering why introduce _ to the language at all? Why not make
that a mechanic external to the language, to the configurations and the things
that setup the variable resolver?
---
[GitHub] metron pull request #1021: METRON-1568: Stellar should have a _ special vari...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1021#discussion_r189907609
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
---
@@ -142,8 +143,14 @@ else if(kv.getValue() instanceof List) {
{
--- End diff --
or - do what you are doing, but just support named maps and use some
default name for the message itself.
It feels like '_' conflates the 'messaging' with the language. ( I say
feels, because obviously I'm just expressing an opinion ).
This code makes the message the namespace, but then makes you reference the
namespace with get map instead of just by name... or seems to.
Also, I hope some of these MAAS applications make it into Metron /contrib ;)
---
[GitHub] metron pull request #1021: METRON-1568: Stellar should have a _ special vari...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1021#discussion_r189924019
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
---
@@ -142,8 +143,14 @@ else if(kv.getValue() instanceof List) {
{
--- End diff --
I'm trying to wrap my head around why you need the special handling in the
variable resolver.
If one of the vars points to a map, that is it. You don't need to do any
special dance.
You can have the variable context 'builder' ( config whatever ) allow
naming the message as a var with a certain name and that is it.
I don't understand why all the extra stuff is required. IE. why the
variable resolver needs to know it is a map. It is just a var.
---
[GitHub] metron pull request #1021: METRON-1568: Stellar should have a _ special vari...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1021#discussion_r189905091
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
---
@@ -142,8 +143,14 @@ else if(kv.getValue() instanceof List) {
{
--- End diff --
I don't doubt the need or applicability, I guess I'm saying why not just
have the _ in the configuration side, and just have the scripts reference the
vars by name and not have to MAP_GET()?
That way, the vars are the vars are the vars in Stellar, and the external
configuration can have the option for putting all the message in the var
namespace.
---
[GitHub] metron issue #754: METRON-1184 EC2 Deployment - Updating control_path to acc...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/754 I don't have an EC2 to use unfortunately. @lvets is the only one who answered the call to test. If it doesn't work on the mac, then we need a new jira and proposed fix I guess ---
[GitHub] metron pull request #1021: METRON-1568: Stellar should have a _ special vari...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1021#discussion_r190018295
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
---
@@ -142,8 +143,14 @@ else if(kv.getValue() instanceof List) {
{
--- End diff --
My impression is that you are creating an implicit scope with the message,
maybe that is where we are crossing
---
[GitHub] metron issue #1021: METRON-1568: Stellar should have a _ special variable wh...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1021 @cestella re:assignment. If all assignment is in the language, and _ is either or, then how can we assign to a message field when _ is active? ---
[GitHub] metron pull request #1021: METRON-1568: Stellar should have a _ special vari...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1021#discussion_r190004639
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
---
@@ -142,8 +143,14 @@ else if(kv.getValue() instanceof List) {
{
--- End diff --
Something like that, in leu of literal scopes ( which would have a
hierarchy ) might work out.
I'm not trying to shoot down your idea, I just want to understand it, and
we approach things differently.
I think in the end we want to get to
interface Scope {
Scope getParent()
Resolve()
Get()
Put()
}
before calling a function/lambda
Scope scope = new Scope(stack.peek())
stack.push(scope)
call function(scope)
stack.pop(scope)
---
[GitHub] metron issue #1045: METRON-1594: KafkaWriter is asynchronous and may lose da...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1045 My comment was just about calling out a possible need for more shutdown orchestration. I am not reviewing. ---
[GitHub] metron issue #1045: METRON-1594: KafkaWriter is asynchronous and may lose da...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1045 I assume you are talking to @nickwallen there @mmiklavc ? ---
[GitHub] metron pull request #1054: METRON-1606 Add capability to wrap json message a...
GitHub user ottobackwards opened a pull request:
https://github.com/apache/metron/pull/1054
METRON-1606 Add capability to wrap json message as entity arrays
This PR adds the ability to configure the JSONMap parser to wrap messages
when using JSON Path queries in an entity with an array.
For example, if the json 'document' is
`{"foo": "one"}, {"bar","two"}`
you need to have it wrapped to work with it with jsonpath.
This PR allows this, resulting in
`{"configured name or default of message" : [ {{"foo": "one"},
{"bar","two"}]}`
Tests where added, along with sample data for integration, and a new sample
configuration.
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/metron json-wrap
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1054.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1054
commit e991d8d43865149f957d530624a69539de5d7bb2
Author: Otto Fowler
Date: 2018-06-07T14:52:07Z
METRON-1606 Add capability to wrap json message as entity arrays
---
[GitHub] metron pull request #1064: METRON-1619: Stellar empty collections should be ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1064#discussion_r195931103 --- Diff: metron-stellar/stellar-common/README.md --- @@ -54,6 +54,12 @@ The Stellar language supports the following: * The ability to have parenthesis to make order of operations explicit * User defined functions, including Lambda expressions +### Boolean Expressions + +Similar to python and javascript, empty collections (e.g. `[]` and --- End diff -- Or am i miss understanding the explicit null case? ---
[GitHub] metron pull request #1064: METRON-1619: Stellar empty collections should be ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1064#discussion_r195931024 --- Diff: metron-stellar/stellar-common/README.md --- @@ -54,6 +54,12 @@ The Stellar language supports the following: * The ability to have parenthesis to make order of operations explicit * User defined functions, including Lambda expressions +### Boolean Expressions + +Similar to python and javascript, empty collections (e.g. `[]` and --- End diff -- Missing variables vs. NULL variables... this may be confusing. What this is saying is we support - boolean with true or false - variables that are present but explicitly null or empty - variables that are NOT present ---
[GitHub] metron issue #1063: METRON-1618: Stellar boolean expressions should treat mi...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1063 ```bash [Stellar]>>> foo := unknownvariable [Stellar]>>> foo [Stellar]>>> ``` This is not consistent. In my stellar assign PR, this is why I execute everything in stellar instead of part shell. ---
[GitHub] metron issue #1034: METRON-1580 Release candidate check script requires Bro ...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1034 +1 by inspection, thanks @nickwallen ---
[GitHub] metron issue #1033: METRON-1579: Stellar should return the expression that f...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1033 +1 I will say, that the exception building facilities and builders should be publicly exposed as part of stellar as a follow on. As part of my own antlr work on the syslog stuff, I am kind of fond of being able to allow expansion by deriving new listeners and visitors with separate implementations, the exception builders would be nice to have with that, if you follow. ---
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 Can I ask how we do ++, --, += etc with the := notation? ---
[GitHub] metron pull request #1039: METRON-1588 Migrate storm-kafka-client to 1.2.1
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1039#discussion_r192225526
--- Diff:
metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutRetryExponentialBackoff.java
---
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.TopicPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.lang.Validate;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+/**
+ * Implementation of {@link KafkaSpoutRetryService} using the exponential
backoff formula. The time of the nextRetry is set as follows:
+ * nextRetry = failCount == 1 ? currentTime + initialDelay : currentTime +
delayPeriod*2^(failCount-1)where failCount = 1, 2, 3, ...
+ * nextRetry = Min(nextRetry, currentTime + maxDelay)
+ */
+public class KafkaSpoutRetryExponentialBackoff implements
KafkaSpoutRetryService {
--- End diff --
There should also be something in the NOTICE file
---
[GitHub] metron issue #1045: METRON-1594: KafkaWriter is asynchronous and may lose da...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1045 Maybe we need some kind of orchestration service that you use to shutdown metron without losing things in the pipeline already ---
[GitHub] metron issue #1035: METRON-1581: kill the profiler topology immediately befo...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1035 Do we know why it is not shutting down? What if this results in data loss? I think we need a better understanding of what is happening. ---
[GitHub] metron issue #1041: METRON-1590: validate-jira-for-release script checks out...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1041 I would say that the scripts we have wrt PR's and RC's and Commits should be as common as possible. With the goal that they actually share code down the line. So I would say that the configuration and preference persistence for working dir and other options be done as nick's scripts ---
[GitHub] metron issue #1041: METRON-1590: validate-jira-for-release script checks out...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1041 Right, we are on the same page. That is one of the scripts I was suggesting you ape. ---
[GitHub] metron pull request #1033: METRON-1579: Stellar should return the expression...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1033#discussion_r191080770
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java
---
@@ -92,6 +96,19 @@ public String getStreamSubGroup(String enrichmentType,
String field) {
}
}
--- End diff --
I think this should be part of stellar itself and not part of enrichment.
---
[GitHub] metron pull request #1033: METRON-1579: Stellar should return the expression...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1033#discussion_r191080813
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/BaseStellarProcessor.java
---
@@ -143,7 +143,11 @@ public T parse(final String rule, final
VariableResolver variableResolver, final
try {
return clazz.cast(expression
.apply(new StellarCompiler.ExpressionState(context,
functionResolver, variableResolver)));
-}finally {
+}
--- End diff --
Maybe we can have stellar exception builders that provide standard,
consistent exception building
---
[GitHub] metron pull request #1083: METRON-1643: Create a REGEX_ROUTING field transfo...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1083#discussion_r198530068 --- Diff: metron-platform/metron-parsers/README.md --- @@ -337,6 +337,28 @@ The following config will rename the fields `old_field` and `different_old_field ] } ``` +* `REGEX_ROUTING` : This transformation lets users set an output field to one of a set of possibilities based on matching regexes. This has functional overlap with Stellar match statements, but the syntax is more bearable for large sets of conditionals. --- End diff -- I don't think we need to editorialize so much about the match statements here. "This transformation is useful when the number or conditions are large enough to make a stellar language match statement unwieldy" would be better. ---
[GitHub] metron issue #1083: METRON-1643: Create a REGEX_ROUTING field transformation
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1083 I am not sure ROUTING is a good name for this. This is more like a SELECT. ---
[GitHub] metron issue #1083: METRON-1643: Create a REGEX_ROUTING field transformation
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1083 +1 to SELECT What you are saying is SET FIELD to X if SELECT. It would be SWITCH if it were a different X per matching regex ---
[GitHub] metron issue #1064: METRON-1619: Stellar empty collections should be conside...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1064 +1 by inspection ---
[GitHub] metron pull request #1083: METRON-1643: Create a REGEX_ROUTING field transfo...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1083#discussion_r198536558 --- Diff: metron-platform/metron-parsers/README.md --- @@ -337,6 +337,28 @@ The following config will rename the fields `old_field` and `different_old_field ] } ``` +* `REGEX_ROUTING` : This transformation lets users set an output field to one of a set of possibilities based on matching regexes. This has functional overlap with Stellar match statements, but the syntax is more bearable for large sets of conditionals. --- End diff -- I don't take it as an indictment on match, but more on stellar and the requirement for a meta programming mode through the transformation construct. So... don't be so tough on yourself. ---
[GitHub] metron issue #1084: METRON-1644: Support parser chaining (NOTE: review METRO...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1084 This is significant enough that I think some level of design write-up is warranted. At some point we'll want to update the top level doc's and diagrams, but I'm OK with that being a follow on. But some description of the approach and how it works is required for proper review. ---
[GitHub] metron issue #1084: METRON-1644: Support parser chaining (NOTE: review METRO...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1084 Yes. The use case is useful, but this is more dev. focused, if that makes sense. ---
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 thanks again @jameslamb! ---
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 can one of you ( @cestella or @merrimanr ) merge? I can't right now ---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206625671
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/PcapServiceImpl.java
---
@@ -199,6 +208,37 @@ public InputStream getRawPcap(String username, String
jobId, Integer page) throw
return inputStream;
}
+ @Override
+ public Map getConfiguration(String username, String
jobId) throws RestException {
+Map configuration = new HashMap<>();
+try {
+ Map jobConfiguration = jobManager.getJob(username,
jobId).getConfiguration();
+ configuration.put(PcapOptions.BASE_PATH.getKey(),
PcapOptions.BASE_PATH.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.FINAL_OUTPUT_PATH.getKey(),
PcapOptions.FINAL_OUTPUT_PATH.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.START_TIME_MS.getKey(),
PcapOptions.START_TIME_MS.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.END_TIME_MS.getKey(),
PcapOptions.END_TIME_MS.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.NUM_REDUCERS.getKey(),
PcapOptions.NUM_REDUCERS.get(jobConfiguration, Integer.class));
+
+ boolean isFixedFilter =
PcapOptions.FILTER_IMPL.get(jobConfiguration, PcapFilterConfigurator.class)
instanceof FixedPcapFilter.Configurator;
+ if (isFixedFilter) {
+configuration.put(FixedPcapOptions.IP_SRC_ADDR.getKey(),
FixedPcapOptions.IP_SRC_ADDR.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_DST_ADDR.getKey(),
FixedPcapOptions.IP_DST_ADDR.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_SRC_PORT.getKey(),
FixedPcapOptions.IP_SRC_PORT.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_DST_PORT.getKey(),
FixedPcapOptions.IP_DST_PORT.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.PROTOCOL.getKey(),
FixedPcapOptions.PROTOCOL.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.PACKET_FILTER.getKey(),
FixedPcapOptions.PACKET_FILTER.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.INCLUDE_REVERSE.getKey(),
FixedPcapOptions.INCLUDE_REVERSE.get(jobConfiguration, String.class));
+ } else {
+configuration.put(QueryPcapOptions.QUERY.getKey(),
QueryPcapOptions.QUERY.get(jobConfiguration, String.class));
+ }
+} catch (JobNotFoundException e) {
+ // do nothing and return null pcapStatus
--- End diff --
We should log here.
---
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 @merrimanr I'd like to get your sign off on this, now that @cestella and I have given a +1 ---
[GitHub] metron pull request #1175: METRON-1453 Metron Parser for valid RFC 5424 Sysl...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1175 METRON-1453 Metron Parser for valid RFC 5424 Syslog messages This is a simple parser for *valid* [RFC 5424](http://www.rfc-base.org/txt/rfc-5424.txt) messages. It produces JSON for the messages, including support for structured data. All structured data will be flattened. It's configuration allows for specifying how missing fields are handled ( "-" in the spec ). They can be: - DASH ( the default and same as spec) - OMIT : the field will not be in the return set - NULL : the field will be in the return set, but as null The parser uses the [simple-syslog-5424](https://github.com/palindromicity/simple-syslog-5424) library. Testing: - Build should work ;) - in full dev, parser configuration should be available - if you have a syslog 5424 compliant data source, hook it up In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [-] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron syslog-5424 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1175.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1175 commit ea2ee761d45049395b585c72e44b406f32cbf881 Author: Otto Fowler Date: 2018-08-24T20:50:30Z METRON-1453 Metron Parser for valid RFC 5424 Syslog messages ---
[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1175#discussion_r213051917
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+ public static final String NIL_POLICY_CONFIG = "nilPolicy";
+ /**
+ * The NilPolicy specifies how the parser handles missing fields in the
return
+ * It can:
+ * Omit the fields
+ * Have a value of '-' ( as spec )
+ * Have null values for the fields
+ * The default is to omit the fields from the return set.
+ */
+ private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+ @Override
+ public void configure(Map config) {
+String nilPolicyStr = (String)
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+ }
+
+ @Override
+ public void init() {
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public List parse(byte[] rawMessage) {
+try {
+ if (rawMessage == null || rawMessage.length == 0) {
+return null;
+ }
+
+ String originalString = new String(rawMessage);
+
+ SyslogParser parser = new
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --
yeah
---
[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1175#discussion_r213016887
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+ public static final String NIL_POLICY_CONFIG = "nilPolicy";
+ /**
+ * The NilPolicy specifies how the parser handles missing fields in the
return
+ * It can:
+ * Omit the fields
+ * Have a value of '-' ( as spec )
+ * Have null values for the fields
+ * The default is to omit the fields from the return set.
+ */
+ private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+ @Override
+ public void configure(Map config) {
+String nilPolicyStr = (String)
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+ }
+
+ @Override
+ public void init() {
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public List parse(byte[] rawMessage) {
+try {
+ if (rawMessage == null || rawMessage.length == 0) {
+return null;
+ }
+
+ String originalString = new String(rawMessage);
+
+ SyslogParser parser = new
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --
You ok with that as part of this pr?
---
[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1175#discussion_r213039514
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+ public static final String NIL_POLICY_CONFIG = "nilPolicy";
+ /**
+ * The NilPolicy specifies how the parser handles missing fields in the
return
+ * It can:
+ * Omit the fields
+ * Have a value of '-' ( as spec )
+ * Have null values for the fields
+ * The default is to omit the fields from the return set.
+ */
+ private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+ @Override
+ public void configure(Map config) {
+String nilPolicyStr = (String)
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+ }
+
+ @Override
+ public void init() {
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public List parse(byte[] rawMessage) {
+try {
+ if (rawMessage == null || rawMessage.length == 0) {
+return null;
+ }
+
+ String originalString = new String(rawMessage);
+
+ SyslogParser parser = new
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --
I don't think I want to touch all the parsers for this PR , there might be
more than one parser follow on depending on how many have configurations. Can
you create a jira or shall I?
---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 All that being said I am a big +1 on this. Great work @justinleet, thanks for taking the time to work it through my thick skull. ---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 A mechanism for the routing process to apply a transform or some such. @cestella may have a better design idea. What I would like us to do is remove the transport from the message where there are common wrappers. Example: All the message types delivered by syslog. The parsers should not have to all parser syslog AND the message. I imagine defining a transform/parser in the router that takes every message and transforms it ( parses syslog fields + structured data if 5424 into meta and MSG to the bytes ) and then passes it on to a parser that only needs to know the message. That way we can have simpler parsers, and even support the same message when transported in different wrappers. We can talk about if this is a function of parser chaining, such that each specialized parser will be second in a chain with the syslog unwrapped being first, or part of routing, but I think it is part of routing personally. ---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 @justinleet I am fine with that as a follow on, I would like the task or issue created. ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r203083284 --- Diff: use-cases/parser_chaining/README.md --- @@ -233,3 +233,10 @@ cat ~/data.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --b ``` You should see indices created for the `cisco-5-304` and `cisco-6-302` data with appropriate fields created for each type. + +# Aggregated Parsers with Parser Chaining +Chained parsers can be run as aggregated parsers. These parsers continue to use the sensor specific Kafka topics, and do not do internal routing to the appropriate sensor. + --- End diff -- yes. I'm not sure we need to document the UI or somewhere else? Do we have to document removing the default processors from ambari if you are going to aggregate them etc? Or how to do it in the management ui? We may need more practical steps. ---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 Ok @justinleet thanks for the diagram. That really helps. I did not see in the code how we were sending out to the sensor topic and then into the sensor, I though the bolt was just calling the parser ( which makes the sensor specific topic not make sense ). I think that this flow is a valid case, one where the data could be coming from either an aggregated flow into the topic *or* straight to the topic. My question looking at this, is why ( esp when people are asking about reduction of kafka overhead ) can't we have the *option* to eliminate the sensor topic completely? ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r203095632 --- Diff: use-cases/parser_chaining/README.md --- @@ -233,3 +233,10 @@ cat ~/data.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --b ``` You should see indices created for the `cisco-5-304` and `cisco-6-302` data with appropriate fields created for each type. + +# Aggregated Parsers with Parser Chaining +Chained parsers can be run as aggregated parsers. These parsers continue to use the sensor specific Kafka topics, and do not do internal routing to the appropriate sensor. + --- End diff -- I think adding the follow on is a great idea. I wonder if we shouldn't change the default install to aggregate the default sensors? I don't think that will work though, because the comma separation in ambari is the list, and you won't be able to do it. My main concern is that the reviewers and committers of this pr are going to be the only ones who can do it, and everyone in user land is going to be lost. If this is going to be expert only ( until the UI comes ) and not recommend, or a preview thing, maybe mark it as such. ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1099#discussion_r202802349
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -182,40 +185,61 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
super.prepare(stormConf, context, collector);
messageGetStrategy = MessageGetters.DEFAULT_BYTES_FROM_POSITION.get();
this.collector = collector;
-if(getSensorParserConfig() != null) {
- cache =
CachingStellarProcessor.createCache(getSensorParserConfig().getCacheConfig());
-}
-initializeStellar();
-if(getSensorParserConfig() != null && filter == null) {
-
getSensorParserConfig().getParserConfig().putIfAbsent("stellarContext",
stellarContext);
- if
(!StringUtils.isEmpty(getSensorParserConfig().getFilterClassName())) {
-filter = Filters.get(getSensorParserConfig().getFilterClassName()
-, getSensorParserConfig().getParserConfig()
-);
+
+// Build the Stellar cache
+Map cacheConfig = new HashMap<>();
+for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ SensorParserConfig config = getSensorParserConfig(sensor);
+
+ if (config != null) {
+cacheConfig.putAll(config.getCacheConfig());
}
}
+cache = CachingStellarProcessor.createCache(cacheConfig);
-parser.init();
+// Need to prep all sensors
+for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ MessageParser parser =
entry.getValue().getMessageParser();
--- End diff --
I'm still not sure sharing the stellar Context is a good thing. @cestella
---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r202797418 --- Diff: metron-platform/metron-parsers/README.md --- @@ -82,6 +82,12 @@ topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an `error` queue. Invalid messages as determined by global validation functions are also treated as errors and sent to an `error` queue. + +Multiple sensors can be aggregated into a single Storm topology. When this is done, there will be +multiple Kafka spouts, but only a single parser bolt which will handle delegating to the correct --- End diff -- There is another, more likely use case where we have a transport wrapper on another message, and 1 topic split into many parsers as well. How can we handle that? Specifically -> Syslog (Many Msg types) -> kafka -> bolt -> Split per message I expect to add the ability for syslog parsing later, so set that aside. The issue is we *will* have more than one use case wrt topics. I am not going to say this PR needs to address it, but I would want us to understand our path forward and minimize the churn. It would be best if we did not have to redo this work when accounting for that. ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r202803106 --- Diff: metron-platform/metron-parsers/README.md --- @@ -82,6 +82,12 @@ topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an `error` queue. Invalid messages as determined by global validation functions are also treated as errors and sent to an `error` queue. + +Multiple sensors can be aggregated into a single Storm topology. When this is done, there will be +multiple Kafka spouts, but only a single parser bolt which will handle delegating to the correct --- End diff -- I think I'm misunderstanding between your diagram and this implementation. There will be one kafka topic monitored by the bolt, then routed to the right parser, then output to a different spout per parser? ---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 @justinleet the main things I saw that I would think of cutting down, or I though about looking into ( the idea may turn out to be bad ) are places where the bolt 'knows' a lot of weird or complicated initialization logic around the configurations or classes it uses, like what we do initializing Stellar, or in getComponentConfiguration. I'm ok with a follow on, I don't think this PR is creating that issue. ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r202755740 --- Diff: metron-platform/metron-parsers/README.md --- @@ -82,6 +82,12 @@ topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an `error` queue. Invalid messages as determined by global validation functions are also treated as errors and sent to an `error` queue. + +Multiple sensors can be aggregated into a single Storm topology. When this is done, there will be +multiple Kafka spouts, but only a single parser bolt which will handle delegating to the correct --- End diff -- So, this means that there is a kafka topic/spout per parser? ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r202808756 --- Diff: metron-platform/metron-parsers/README.md --- @@ -82,6 +82,12 @@ topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an `error` queue. Invalid messages as determined by global validation functions are also treated as errors and sent to an `error` queue. + +Multiple sensors can be aggregated into a single Storm topology. When this is done, there will be +multiple Kafka spouts, but only a single parser bolt which will handle delegating to the correct --- End diff -- The bolt is actually executing the parser, not sending it to kafka though. Let's say that I route all my syslog stuff ( multiple message types ) through 1 kafka topic ( not tied to _ANY_ sensor. I would want to point this bolt at that one topic, and have it route to multiple parsers. I think then they all just go to enrichment? ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r202812681 --- Diff: metron-platform/metron-parsers/README.md --- @@ -82,6 +82,12 @@ topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an `error` queue. Invalid messages as determined by global validation functions are also treated as errors and sent to an `error` queue. + +Multiple sensors can be aggregated into a single Storm topology. When this is done, there will be +multiple Kafka spouts, but only a single parser bolt which will handle delegating to the correct --- End diff -- what *is* the intermediate kafka step? That is what is confusing me. My understanding is that for each sensor you reference it will build a spout for that sensor parser topic, and then pass everything from those to this bolt, which will call the right parser and output to I'm not sure. Why have to have a sensor specific topic at all? ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1099#discussion_r202758396
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -182,40 +185,61 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
super.prepare(stormConf, context, collector);
messageGetStrategy = MessageGetters.DEFAULT_BYTES_FROM_POSITION.get();
this.collector = collector;
-if(getSensorParserConfig() != null) {
- cache =
CachingStellarProcessor.createCache(getSensorParserConfig().getCacheConfig());
-}
-initializeStellar();
-if(getSensorParserConfig() != null && filter == null) {
-
getSensorParserConfig().getParserConfig().putIfAbsent("stellarContext",
stellarContext);
- if
(!StringUtils.isEmpty(getSensorParserConfig().getFilterClassName())) {
-filter = Filters.get(getSensorParserConfig().getFilterClassName()
-, getSensorParserConfig().getParserConfig()
-);
+
+// Build the Stellar cache
+Map cacheConfig = new HashMap<>();
+for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ SensorParserConfig config = getSensorParserConfig(sensor);
+
+ if (config != null) {
+cacheConfig.putAll(config.getCacheConfig());
}
}
+cache = CachingStellarProcessor.createCache(cacheConfig);
-parser.init();
+// Need to prep all sensors
+for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ MessageParser parser =
entry.getValue().getMessageParser();
--- End diff --
I don't believe this is correct. We want to initialize stellar PER parser.
Each should have it's own stellar instance and cache.
---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1099#discussion_r202798006
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -182,40 +185,61 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
super.prepare(stormConf, context, collector);
messageGetStrategy = MessageGetters.DEFAULT_BYTES_FROM_POSITION.get();
this.collector = collector;
-if(getSensorParserConfig() != null) {
- cache =
CachingStellarProcessor.createCache(getSensorParserConfig().getCacheConfig());
-}
-initializeStellar();
-if(getSensorParserConfig() != null && filter == null) {
-
getSensorParserConfig().getParserConfig().putIfAbsent("stellarContext",
stellarContext);
- if
(!StringUtils.isEmpty(getSensorParserConfig().getFilterClassName())) {
-filter = Filters.get(getSensorParserConfig().getFilterClassName()
-, getSensorParserConfig().getParserConfig()
-);
+
+// Build the Stellar cache
+Map cacheConfig = new HashMap<>();
+for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ SensorParserConfig config = getSensorParserConfig(sensor);
+
+ if (config != null) {
+cacheConfig.putAll(config.getCacheConfig());
}
}
+cache = CachingStellarProcessor.createCache(cacheConfig);
-parser.init();
+// Need to prep all sensors
+for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ MessageParser parser =
entry.getValue().getMessageParser();
--- End diff --
We may need a test then.
---
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1112 or, maybe we are just missing each other here, and you can explain how the user will sign on. SSO doesn't mean no sign on. How will I now provide my user name and password in the app? ---
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1112 this might be worth a discuss thread @simonellistonball ---
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1112 I don't understand, how are you going to do the auth without the login screen? ---
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1112 "The authentication will be handled by the hosts that allow loading of the UIs redirecting the browser to a KnoxSSO endpoint, handled in METRON-1665" How is this going to work in vagrant? What will be the new minimum required setup for security to use Metron's UI's now? ---
[GitHub] metron issue #865: METRON-1212 The bundle System and Maven Plugin (Feature B...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/865 ok, i give up ---
[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/865 ---
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1112 @simonellistonball, thank you. I didn't get that from the PR description. Sorry for the noise. ---
[GitHub] metron issue #1103: METRON-1554: Initial PCAP UI
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1103 I think we should rename from alert ui to investigate or something ---
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 Great! I will give this a try asap ---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 I have been on vacation, but will be reviewing Monday and Tuesday. Please do not commit ---
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 +1 from me. I was able to do the above, along with building metron from the instructions ansible-docker's readme.md. Thanks for sticking with it. ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r203064655 --- Diff: use-cases/parser_chaining/README.md --- @@ -233,3 +233,10 @@ cat ~/data.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --b ``` You should see indices created for the `cisco-5-304` and `cisco-6-302` data with appropriate fields created for each type. + +# Aggregated Parsers with Parser Chaining +Chained parsers can be run as aggregated parsers. These parsers continue to use the sensor specific Kafka topics, and do not do internal routing to the appropriate sensor. + --- End diff -- Should we explicitly say that when you use this type of topology so do NOT create a sensor topology for any sensor included in the aggregate? ---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 Sure, actually I'll do a discuss thread when this all goes through. That way I can try again to get @cestella to comment ---
[GitHub] metron issue #1178: METRON-1757 Storm Profiler Serialization Exception
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1178 The work-around to this issue, and some documentation of it to the extent you feel necessary should go out to the users list. ---
[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1175#discussion_r213706134
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+ public static final String NIL_POLICY_CONFIG = "nilPolicy";
+ private transient SyslogParser syslogParser;
+
+ @Override
+ public void configure(Map config) {
+// Default to OMIT policy for nil fields
+// this means they will not be in the returned field set
+String nilPolicyStr = (String)
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+NilPolicy nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+syslogParser = new
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
+ }
+
+ @Override
+ public void init() {
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public List parse(byte[] rawMessage) {
+try {
+ if (rawMessage == null || rawMessage.length == 0) {
+return null;
+ }
+
+ String originalString = new String(rawMessage);
+ JSONObject jsonObject = new
JSONObject(syslogParser.parseLine(originalString));
+
+ // be sure to put in the original string, and the timestamp.
+ // we wil just copy over the timestamp from the syslog
+ jsonObject.put("original_string", originalString);
+ jsonObject.put("timestamp",
jsonObject.get(SyslogFieldKeys.HEADER_TIMESTAMP.getField()));
--- End diff --
Great point, latest commit has the change and the test
---
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1184 METRON-1761, allow application of grok statement multiple times This PR adds support for incoming messages to grok parsers that have multiple log lines. Instead of having to split the logs before sending to metron/kafka, you could just send the logs in batches. # todo testing ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [-] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron grok-split Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1184.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1184 commit 5cb7cb2b869694ceff37c7e07e16358e4b918b2c Author: Otto Fowler Date: 2018-09-04T10:50:38Z METRON-1761, allow application of grok statement multiple times ---
[GitHub] metron-bro-plugin-kafka issue #8: METRON-1768: Adjust versioning of metron-b...
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/8 +1 ---
[GitHub] metron issue #1175: METRON-1750 Metron Parser for valid RFC 5424 Syslog mess...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1175 New upstream integrated now. ---
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 @jameslamb, The vagrant stuff is the general information on how to run it. For the docker containers that you have worked on, it would be fine I think to say that you tested that the containers work for their purpose. It would be enough for you to say in the description: ``` Testing: run ansible docker like this %THIS, see that it works without error run deb docker like this %THIS, see that it works without error run rpm docker like this %THIS, see that it works without error ``` Our vagrant build for centos and ubuntu targets actually use the rpm and deb dockers so that is one way they get tests, but doesn't have to be the only way. the ansible docker must be tested explicitly ---
[GitHub] metron issue #1084: METRON-1644: Support parser chaining
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1084 1 nit, other than that +1 ---
[GitHub] metron pull request #1084: METRON-1644: Support parser chaining
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1084#discussion_r201426118
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/message/metadata/EnvelopedRawMessageStrategy.java
---
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.message.metadata;
+
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.utils.JSONUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * An alternative strategy whereby
+ *
+ * The raw data is presumed to be a JSON Map
+ * The data to be parsed is the contents of one of the fields.
+ * The non-data fields are considered metadata
+ *
+ *
+ * Additionally, the defaults around merging and reading metadata are
adjusted to be on by default.
+ * Note, this strategy allows for parser chaining and for a fully worked
example, check the parser chaining use-case.
+ */
+public class EnvelopedRawMessageStrategy implements RawMessageStrategy {
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ /**
+ * The field from the rawMessageStrategyConfig in the SensorParserConfig
that defines the field to use to
+ * define the data to be parsed.
+ */
+ public static final String MESSAGE_FIELD_CONFIG = "messageField";
+
+ /**
+ * Retrieve the raw message by parsing the JSON Map in the kafka value
and pulling the appropriate field.
+ * Also, augment the default metadata with the non-data fields in the
JSON Map.
+ *
+ * Note: The data field in the JSON Map is not considered metadata.
+ *
+ * @param rawMetadata The metadata read from kafka Key (e.g. the topic,
index, etc.)
+ * @param rawMessage The raw message from the kafka value
+ * @param readMetadata True if we want to read read the metadata
+ * @param config The config for the RawMessageStrategy (See the
rawMessageStrategyConfig in the SensorParserConfig)
+ * @return
+ */
+ @Override
+ public RawMessage get(Map rawMetadata, byte[]
rawMessage, boolean readMetadata, Map config) {
+String messageField = (String)config.get(MESSAGE_FIELD_CONFIG);
+if(messageField == null) {
+ throw new IllegalStateException("You must specify a message field in
the message supplier config. " +
+ "I expected to find a \"messageField\" field in the
config.");
--- End diff --
Not sure about the first person here. I'm all for the @cestella / metron
convergence, but let us not rush into that good night
---
[GitHub] metron issue #1084: METRON-1644: Support parser chaining
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1084 I'm looking for a doc that describes how these new things work, like what would have come out of a discuss thread if there had been discussion on this before hand. " Dev. design notes" "The way this works is ... We use the configuration ago do this, and the new strategies to do that, basically setting the system up so that we can do x,y and z in this component. See this code, that code and this code." "Other ways to do this are foo and bar, but I chose this because cause blah". ---
[GitHub] metron issue #1084: METRON-1644: Support parser chaining
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1084 Let's try something else. Please javadoc all the new classes and functionality, such that someone else if they want to review or maintain this can understand their implementation ---
[GitHub] metron issue #1083: METRON-1643: Create a REGEX_ROUTING field transformation
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1083 " Ultimately, I consider this a stop-gap." Yes. What we are basically doing is writing a meta language on top of stellar. In this case we are using that to make up for the meta language itself not supporting multi-line input ( and stellar's support for that as well ) A question would be, why would this transform not just be transformed into a match statement instead of adding a new function? How much of this pain is from not using the UI for editing? Would a ui editor that let you put in a multi line match make this pain better? ---
[GitHub] metron issue #1083: METRON-1643: Create a REGEX_ROUTING field transformation
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1083 Why don't you create a jira for the REGEXP_MATCH ---
