Re: [DISCUSS] Mutation of Indexed Data

2017-06-21 Thread Simon Elliston Ball
I'd say that was an excellent set of requirements (very similar to the one we arrived on with the last discuss thread on this) My vote remains a transaction log in hbase given the relatively low volume (human scale) i would not expect this to need anything fancy like compaction into hdfs

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r123420154 --- Diff: metron-platform/metron-extensions/metron-parser-extensions/metron-parser-bro-extension/metron-parser-bro/pom.xml --- @@ -0,0 +1,165 @@

Re: [DISCUSS] Mutation of Indexed Data

2017-06-21 Thread Otto Fowler
Can you clarify what data stores are at play here? On June 21, 2017 at 17:07:42, Casey Stella (ceste...@gmail.com) wrote: Hi All, I know we've had a couple of these already, but we're due for another discussion of a sensible approach to mutating indexed data. The motivation for this is users

Re: [DISCUSS] Metadata Ingest

2017-06-21 Thread Simon Elliston Ball
I really like this idea. A good use case I imagine would be to have something like asa data, tagged with some custom meta data (e.g. Tenant ID in a multi-tenant install) but not have to mess with the actual parser. To that extent it makes sense to expose said meta data via stellar so users can

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r123415116 --- Diff: metron-platform/metron-extensions/metron-parser-extensions/metron-parser-bro-extension/metron-parser-bro/pom.xml --- @@ -0,0 +1,165 @@

Re: [DISCUSS] Metadata Ingest

2017-06-21 Thread Otto Fowler
First: Thanks Casey. I submitted a review in the PR, that I will not duplicate here. I would say however the following: - I would like to understand the problem we are trying to solve with this more. This seems like a good idea, and a capability we obviously can imagine how to implement, but

[GitHub] metron pull request #621: METRON-1001: Allow metron to ingest parser metadat...

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/621#discussion_r123412506 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java --- @@ -118,6 +116,27 @@ protected void

[GitHub] metron pull request #621: METRON-1001: Allow metron to ingest parser metadat...

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/621#discussion_r123411616 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java --- @@ -118,6 +116,27 @@ protected void

Re: storm-kafka-client use in Metron

2017-06-21 Thread Casey Stella
Thanks for the heads up, kris. This is really good context and we should be aware of it. So we could avoid this by avoiding subscription objects which rely on consumer.subscribe and prefer consumer.assign, correct? On Tue, Jun 20, 2017 at 10:07 PM, Kristopher Kane wrote:

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123374060 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -253,69

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123374090 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -253,69

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123373290 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -253,69

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123372371 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -253,69

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123370264 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -253,69

[GitHub] metron issue #528: METRON-838 Incorrect set of ts in FireEye parser

2017-06-21 Thread bjigmp
Github user bjigmp commented on the issue: https://github.com/apache/metron/pull/528 Wrote test for FireEye and found that it uses ParserUtils.convertToEpoch that returns incorrect value. Filed METRON-1003. Will create PR to fix this bug and then continue with this PR. --- If

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123368840 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -253,69

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123368823 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -253,69

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123368910 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -253,69

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123368861 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/enrichment.properties.j2 ---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r123367425 --- Diff: metron-platform/metron-extensions/metron-parser-extensions/metron-parser-bro-extension/metron-parser-bro/pom.xml --- @@ -0,0 +1,165 @@

[DISCUSS] Mutation of Indexed Data

2017-06-21 Thread Casey Stella
Hi All, I know we've had a couple of these already, but we're due for another discussion of a sensible approach to mutating indexed data. The motivation for this is users will want to update fields to correct and augment data. These corrections are invaluable for things like feedback for ML

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r123359217 --- Diff: metron-analytics/metron-maas-service/README.md --- @@ -138,7 +138,7 @@ Now that we have a deployed model, let's adjust the configurations

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 If we want to come up with a different name for the assembly ( but it will still be tgz ) I would be open to that. I did not want to go that far without review --- If your project is set up

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 I will try to come up with a diagram about the assembly composition with a dictionary --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r123356422 --- Diff: metron-platform/metron-extensions/metron-parser-extensions/metron-parser-bro-extension/metron-parser-bro/pom.xml --- @@ -0,0 +1,165 @@

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r123356137 --- Diff: bundles-maven-plugin/src/main/resources/META-INF/plexus/components.xml --- @@ -0,0 +1,52 @@ + + + + + +

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r123356168 --- Diff: bundles-maven-plugin/src/main/java/org/apache/metron/BundleMojo.java --- @@ -0,0 +1,741 @@ +/* + * Licensed to the Apache Software

[DISCUSS] Metadata Ingest

2017-06-21 Thread Casey Stella
Hi All, I wanted to call attention to a JIRA (METRON-1001) that I just submitted and possibly discuss it more broader than on the PR. Currently, we only ingest data in Metron. Often, there is valuable metadata constructed up-stream of Metron that is relevant to enrichment and cross-cuts many

[GitHub] metron pull request #621: METRON-1001: Allow metron to ingest parser metadat...

2017-06-21 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/621 METRON-1001: Allow metron to ingest parser metadata along with data ## Contributor Comments Currently, we only ingest data in Metron. Often, there is valuable metadata constructed up-stream of

[GitHub] metron issue #608: METRON-986 Enhance Fastcapa to Support Intel X520

2017-06-21 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/608 Did I answer your question @cestella ? Short answer is no and unfortunately I don't quite know how to get there. --- If your project is set up for it, you can reply to this email and have your

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r123291122 --- Diff: metron-analytics/metron-maas-service/README.md --- @@ -138,7 +138,7 @@ Now that we have a deployed model, let's adjust the configurations

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123290700 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -253,69

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123287744 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -253,69

[GitHub] metron issue #586: METRON-508 Expand Elasticsearch templates to support the ...

2017-06-21 Thread JonZeolla
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/586 @nickwallen So, I'm not entirely done with the documentation but I pushed it out for a quick, general review. In doing this, I noticed that some new default-on fields were added with the release

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r123285499 --- Diff: metron-platform/metron-extensions/metron-parser-extensions/metron-parser-bro-extension/metron-parser-bro-assembly/src/main/assembly/assembly.xml

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123263841 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -253,69

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123265664 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -253,69

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-21 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/613#discussion_r123267781 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/enrichment.properties.j2 ---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

2017-06-21 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r123250233 --- Diff: bundles-maven-plugin/.gitignore --- @@ -0,0 +1,16 @@ +target +.project +.settings +.classpath +nbactions.xml

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-06-21 Thread iraghumitra
Github user iraghumitra commented on the issue: https://github.com/apache/metron/pull/620 @merrimanr I missed escaping values for the search request. I fixed it now, the 'referer' search should work now. Supporting 'OR' and 'NOT' operators in search request needs some work is it fine

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

2017-06-21 Thread iraghumitra
Github user iraghumitra closed the pull request at: https://github.com/apache/metron/pull/620 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

2017-06-21 Thread iraghumitra
GitHub user iraghumitra reopened a pull request: https://github.com/apache/metron/pull/620 Metron-988: UI for viewing alerts generated by Metron ## Contributor Comments ## Description Metron being a cybersecurity framework has the capability to raise alerts based