Re: [DISCUSS] Community Meetings

2017-12-13 Thread Simon Elliston Ball
Good points Larry, we would need to get consent from everyone on the call to record to properly comply with regulations in some countries. We would definitely need someone to step up as note taker. Something else to think about is intended audience. Previously we’ve had meeting like this

[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...

2017-12-13 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r156674159 --- Diff: metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java --- @@ -229,17 +239,30 @@ public void

[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...

2017-12-13 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r156675356 --- Diff: metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java --- @@ -229,17 +239,30 @@ public void

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

2017-12-13 Thread ottobackwards
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/865 METRON-1212 The bundle System and Maven Plugin (Feature Branch) This PR contains the Bundle system and Maven Plugin. The bundle system and the plugin are adapted from the Apache Nifi

[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...

2017-12-13 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r156676155 --- Diff: metron-platform/metron-indexing/README.md --- @@ -15,6 +15,12 @@ Indices are written in batch and the batch size and batch timeout are specified

[GitHub] metron pull request #774: METRON-1212 The bundle system and maven plugin

2017-12-13 Thread ottobackwards
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/774 ---

Re: [DISCUSS] Community Meetings

2017-12-13 Thread Otto Fowler
I am ok with just notes and no recording. On December 13, 2017 at 04:37:20, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Good points Larry, we would need to get consent from everyone on the call to record to properly comply with regulations in some countries. We would definitely

[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...

2017-12-13 Thread simonellistonball
Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r156676868 --- Diff: metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java --- @@ -229,17 +239,30 @@ public

[GitHub] metron issue #863: METRON-1347: Indexing Topology should fail tuples without...

2017-12-13 Thread merrimanr
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/863 I would like to hear feedback from @ottobackwards on other required fields but this looks good to me otherwise. ---

[GitHub] metron issue #863: METRON-1347: Indexing Topology should fail tuples without...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/863 The minimum required fields, as far as I can see right now are source.type, original_string and timestamp. Given the use case for this is something that has skipped the parser topology, we

[GitHub] metron issue #862: METRON-1343: Swagger UI for User Controller needs request...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/862 +1, Thanks for the contribution! ---

Re: [DISCUSS] Integration/e2e test infrastructure requirements

2017-12-13 Thread Ryan Merriman
I took a first pass at adding tasks and will continue adding more as I think of them. I will wait for feedback on which modules to include before I add all those (only added metron-elasticsearch for now). I left all but a couple unassigned so that anyone can pick up a task if they want. On Wed,

Re: [DISCUSS] Integration/e2e test infrastructure requirements

2017-12-13 Thread Otto Fowler
Awesome Ryan! Have you thought about confluence? On December 13, 2017 at 18:11:39, Ryan Merriman (merrim...@gmail.com) wrote: I took a first pass at adding tasks and will continue adding more as I think of them. I will wait for feedback on which modules to include before I add all those (only

[GitHub] metron issue #831: METRON-1302: Split up Indexing Topology into batch and ra...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/831 The batch v. hdfs stuff still confuses me, I thought we decided on a different name? ---

[GitHub] metron pull request #866: METRON-1349 Full Dev Builds Metron Twice

2017-12-13 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/866#discussion_r156726426 --- Diff: metron-deployment/playbooks/metron_install.yml --- @@ -15,13 +15,6 @@ # limitations under the License. # --- -- hosts:

[GitHub] metron pull request #866: METRON-1349 Full Dev Builds Metron Twice

2017-12-13 Thread nickwallen
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/866 METRON-1349 Full Dev Builds Metron Twice Removing the "Quick Dev" environment in #852 had an unintended side effect. It caused Metron to be built twice during the Full Dev deployment process.

[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts

2017-12-13 Thread mmiklavc
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/857 Follow up from @merrimanr and my work yesterday. We upped the versions of Node to 9.2.1. Per the doc, >8 is required to work with async/await. For good measure, I also set the NPM version to 5.6.0.

[GitHub] metron pull request #862: METRON-1343: Swagger UI for User Controller needs ...

2017-12-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/862 ---

[GitHub] metron issue #862: METRON-1343: Swagger UI for User Controller needs request...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/862 Please take care to mark the jira as done ---

[GitHub] metron pull request #866: METRON-1349 Full Dev Builds Metron Twice

2017-12-13 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/866#discussion_r156725657 --- Diff: metron-deployment/roles/ambari_config/tasks/main.yml --- @@ -26,16 +26,15 @@ retries: 5 delay: 10 -- name : check if

[GitHub] metron issue #863: METRON-1347: Indexing Topology should fail tuples without...

2017-12-13 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/metron/pull/863 Actually, I don't think `original_string` is required past the parser topology. For instance, profiler messages into enrichment do not have `original_string`. ---

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

2017-12-13 Thread mmiklavc
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/859#discussion_r156723676 --- Diff: metron-deployment/roles/ambari_config/vars/small_cluster.yml --- @@ -87,6 +87,8 @@ configurations: topology.classpath: '{{

[GitHub] metron pull request #866: METRON-1349 Full Dev Builds Metron Twice

2017-12-13 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/866#discussion_r156724938 --- Diff: metron-deployment/roles/epel/tasks/main.yml --- @@ -16,6 +16,4 @@ # --- - name: Install EPEL repository - yum:

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/859#discussion_r156751241 --- Diff: metron-deployment/amazon-ec2/README.md --- @@ -126,6 +126,10 @@ To provision only subsets of the entire Metron deployment, Ansible tags can

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/859#discussion_r156737230 --- Diff: metron-deployment/amazon-ec2/README.md --- @@ -126,6 +126,10 @@ To provision only subsets of the entire Metron deployment, Ansible tags can

[GitHub] metron pull request #866: METRON-1349 Full Dev Builds Metron Twice

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/866#discussion_r156737918 --- Diff: metron-deployment/playbooks/metron_install.yml --- @@ -15,13 +15,6 @@ # limitations under the License. # --- -- hosts:

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

2017-12-13 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/859#discussion_r156744095 --- Diff: metron-deployment/amazon-ec2/README.md --- @@ -126,6 +126,10 @@ To provision only subsets of the entire Metron deployment, Ansible tags can be s

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

2017-12-13 Thread mmiklavc
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/859#discussion_r156748130 --- Diff: metron-deployment/amazon-ec2/README.md --- @@ -126,6 +126,10 @@ To provision only subsets of the entire Metron deployment, Ansible tags can be s

Metron - Emailing Alerts

2017-12-13 Thread Ahmed Shah
Hello, Just wondering if Metron has a feature to email alerts based on rules that a user defines. Example: Rule A: Email the user 1...@1.com whenever ip_src_addr=100.2.10.* Rule B: Email the user 1...@1.com whenever payload contains "critical" If not, does anyone have any recommendations on

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/859#discussion_r156755792 --- Diff: metron-deployment/amazon-ec2/README.md --- @@ -126,6 +126,10 @@ To provision only subsets of the entire Metron deployment, Ansible tags can

Re: Metron - Emailing Alerts

2017-12-13 Thread Simon Elliston Ball
Metron generates alerts onto a Kafka queue, which can be used to integrate with Alert management tools, usually some sort of existing alert aggregation tool. An alternative approach common with this is to have a tool like Apache NiFi attach to the Metron alert feed and send email. The

Re: [DISCUSS] Integration/e2e test infrastructure requirements

2017-12-13 Thread Otto Fowler
What is the Master Jira going to be? On December 13, 2017 at 14:36:50, Ryan Merriman (merrim...@gmail.com) wrote: I am going to start the process of creating Jiras out of these initial requirements. I agree with them and think they are a good starting point. Feel free to join in at anytime and

Re: [DISCUSS] Integration/e2e test infrastructure requirements

2017-12-13 Thread Ryan Merriman
I'm open to ideas. What do you think the title should be? On Wed, Dec 13, 2017 at 2:13 PM, Otto Fowler wrote: > What is the Master Jira going to be? > > > > On December 13, 2017 at 14:36:50, Ryan Merriman (merrim...@gmail.com) > wrote: > > I am going to start the

[GitHub] metron pull request #866: METRON-1349 Full Dev Builds Metron Twice

2017-12-13 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/866#discussion_r156746647 --- Diff: metron-deployment/playbooks/metron_install.yml --- @@ -15,13 +15,6 @@ # limitations under the License. # --- -- hosts:

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/867 METRON-1350: Add reservoir sampling functions to Stellar ## Contributor Comments Sampling capabilities would fit very well with the profiler and enable algorithms that do not necessarily

Re: Metron - Emailing Alerts

2017-12-13 Thread James Sirota
I agree with Simon. If you email each alert individually you will be overwhelmed. I think a better idea would be to email alert summaries periodically, which is more manageable. This is probably a feature worthy of consideration for Metron. 13.12.2017, 12:19, "Simon Elliston Ball"

Re: [DISCUSS] Integration/e2e test infrastructure requirements

2017-12-13 Thread Ryan Merriman
I am going to start the process of creating Jiras out of these initial requirements. I agree with them and think they are a good starting point. Feel free to join in at anytime and add/change/remove requirements as needed. I will update the thread once I have the initial Jiras created and we can

[GitHub] metron issue #867: METRON-1350: Add reservoir sampling functions to Stellar

2017-12-13 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/metron/pull/867 Sorry, I am not sure I understand, this is random replacement when after the size limit. Am I mistaking your question? ---

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156790508 --- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/sampling/SamplingInitFunctions.java --- @@ -0,0 +1,89 @@

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156788055 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. *

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156792227 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. * bounds

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156792855 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. *

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156794655 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. * bounds

Re: Metron - Emailing Alerts

2017-12-13 Thread James Sirota
I think there may be gaps in doing it with the profiler. You can record stats and counts of different alert types, and maybe even alert ids, but you can't cross-correlate these IDs to the alert body. At least not in the profiler. I was thinking about emailing something that looks like a

Re: Metron - Emailing Alerts

2017-12-13 Thread Simon Elliston Ball
That makes a lot of sense, especially if you wanted the detail in the email as well. We could definitely use some good "reporting of alerts” functionality that would make something like that work. What do people think? Simon > On 13 Dec 2017, at 21:52, James Sirota wrote:

Re: Metron - Emailing Alerts

2017-12-13 Thread Otto Fowler
While summary of _any_ metron data ( perhaps by query etc ) would be good, let us not lose sight of the OP’s issue. Ever with summary|digest or one at a time, they are looking for sending mails to certain people based on rule. A pseudo path may be INDEXING -> New Topology or ?? -> evaluate

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156799548 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. *

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156799950 --- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/sampling/UniformSampler.java --- @@ -0,0 +1,91 @@ +/**

Re: Metron - Emailing Alerts

2017-12-13 Thread Simon Elliston Ball
We can already do that with profiles I would have thought. Create a profile that only picks alerts and then base your emails only from the alert events produced by that profile. Would that create the right batching mechanism (at a cost of possible higher latency than you might get with a more

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156790945 --- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/sampling/UniformSampler.java --- @@ -0,0 +1,91 @@ +/** + *

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread simonellistonball
Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156791019 --- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/sampling/UniformSampler.java --- @@ -0,0 +1,91 @@

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156792461 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. *

Re: [DISCUSS] Community Meetings

2017-12-13 Thread James Sirota
I can set up a dedicated Zoom room with a recurrent meeting and give PMC members rights to the room. I think hosting these meetings should not be a problem. I would vote not to record them, but rather provide the notes after the meeting. It's a lot easier to skim through the notes than jump

Re: [DISCUSS] Community Meetings

2017-12-13 Thread Otto Fowler
+1 On December 13, 2017 at 16:39:52, James Sirota (jsir...@apache.org) wrote: I can set up a dedicated Zoom room with a recurrent meeting and give PMC members rights to the room. I think hosting these meetings should not be a problem. I would vote not to record them, but rather provide the

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread simonellistonball
Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156794990 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used.

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156796690 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. * bounds

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156799854 --- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/sampling/UniformSampler.java --- @@ -0,0 +1,91 @@ +/**

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

2017-12-13 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156800428 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. * bounds

Re: Metron - Emailing Alerts

2017-12-13 Thread Otto Fowler
We could also filter out of enrichment to a different topology based on field like Simon has said so that the rules are run on a filtered set etc. also s/Ever/Either/ On December 13, 2017 at 17:03:15, Otto Fowler (ottobackwa...@gmail.com) wrote: While summary of _any_ metron data ( perhaps by

Re: [DISCUSS] Integration/e2e test infrastructure requirements

2017-12-13 Thread Ryan Merriman
Jira is here: https://issues.apache.org/jira/browse/METRON-1352. I am starting to create sub-tasks based on the requirements outlined above and included in that Jira description. I am compiling a list of modules that we'll need to convert to the testing infrastructure. Based on imports of