[GitHub] metron issue #831: METRON-1302: Split up Indexing Topology into batch and ra...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/831 The batch v. hdfs stuff still confuses me, I thought we decided on a different name? ---

Re: [DISCUSS] Integration/e2e test infrastructure requirements

Awesome Ryan! Have you thought about confluence? On December 13, 2017 at 18:11:39, Ryan Merriman (merrim...@gmail.com) wrote: I took a first pass at adding tasks and will continue adding more as I think of them. I will wait for feedback on which modules to include before I add all those (only

Re: [DISCUSS] Integration/e2e test infrastructure requirements

I took a first pass at adding tasks and will continue adding more as I think of them. I will wait for feedback on which modules to include before I add all those (only added metron-elasticsearch for now). I left all but a couple unassigned so that anyone can pick up a task if they want. On Wed,

Re: Metron - Emailing Alerts

We could also filter out of enrichment to a different topology based on field like Simon has said so that the rules are run on a filtered set etc. also s/Ever/Either/ On December 13, 2017 at 17:03:15, Otto Fowler (ottobackwa...@gmail.com) wrote: While summary of _any_ metron data ( perhaps by

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156800428 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. * bounds

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156799950 --- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/sampling/UniformSampler.java --- @@ -0,0 +1,91 @@ +/**

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156799854 --- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/sampling/UniformSampler.java --- @@ -0,0 +1,91 @@ +/**

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156799548 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. *

Re: Metron - Emailing Alerts

While summary of _any_ metron data ( perhaps by query etc ) would be good, let us not lose sight of the OP’s issue. Ever with summary|digest or one at a time, they are looking for sending mails to certain people based on rule. A pseudo path may be INDEXING -> New Topology or ?? -> evaluate

Re: Metron - Emailing Alerts

That makes a lot of sense, especially if you wanted the detail in the email as well. We could definitely use some good "reporting of alerts” functionality that would make something like that work. What do people think? Simon > On 13 Dec 2017, at 21:52, James Sirota wrote:

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156796690 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. * bounds

Re: Metron - Emailing Alerts

I think there may be gaps in doing it with the profiler. You can record stats and counts of different alert types, and maybe even alert ids, but you can't cross-correlate these IDs to the alert body. At least not in the profiler. I was thinking about emailing something that looks like a

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156794990 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used.

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156794655 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. * bounds

Re: [DISCUSS] Community Meetings

+1 On December 13, 2017 at 16:39:52, James Sirota (jsir...@apache.org) wrote: I can set up a dedicated Zoom room with a recurrent meeting and give PMC members rights to the room. I think hosting these meetings should not be a problem. I would vote not to record them, but rather provide the

Re: [DISCUSS] Community Meetings

I can set up a dedicated Zoom room with a recurrent meeting and give PMC members rights to the room. I think hosting these meetings should not be a problem. I would vote not to record them, but rather provide the notes after the meeting. It's a lot easier to skim through the notes than jump

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156792855 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. *

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156792461 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. *

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156788055 --- Diff: metron-analytics/metron-statistics/README.md --- @@ -53,6 +53,32 @@ functions can be used from everywhere where Stellar is used. *

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156790508 --- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/sampling/SamplingInitFunctions.java --- @@ -0,0 +1,89 @@

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156791019 --- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/sampling/UniformSampler.java --- @@ -0,0 +1,91 @@

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/867#discussion_r156790945 --- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/sampling/UniformSampler.java --- @@ -0,0 +1,91 @@ +/** + *

[GitHub] metron issue #867: METRON-1350: Add reservoir sampling functions to Stellar

Github user cestella commented on the issue: https://github.com/apache/metron/pull/867 Sorry, I am not sure I understand, this is random replacement when after the size limit. Am I mistaking your question? ---

Re: Metron - Emailing Alerts

We can already do that with profiles I would have thought. Create a profile that only picks alerts and then base your emails only from the alert events produced by that profile. Would that create the right batching mechanism (at a cost of possible higher latency than you might get with a more

Re: Metron - Emailing Alerts

I agree with Simon. If you email each alert individually you will be overwhelmed. I think a better idea would be to email alert summaries periodically, which is more manageable. This is probably a feature worthy of consideration for Metron. 13.12.2017, 12:19, "Simon Elliston Ball"

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/867 METRON-1350: Add reservoir sampling functions to Stellar ## Contributor Comments Sampling capabilities would fit very well with the profiler and enable algorithms that do not necessarily

Re: [DISCUSS] Integration/e2e test infrastructure requirements

I'm open to ideas. What do you think the title should be? On Wed, Dec 13, 2017 at 2:13 PM, Otto Fowler wrote: > What is the Master Jira going to be? > > > > On December 13, 2017 at 14:36:50, Ryan Merriman (merrim...@gmail.com) > wrote: > > I am going to start the

Re: [DISCUSS] Integration/e2e test infrastructure requirements

What is the Master Jira going to be? On December 13, 2017 at 14:36:50, Ryan Merriman (merrim...@gmail.com) wrote: I am going to start the process of creating Jiras out of these initial requirements. I agree with them and think they are a good starting point. Feel free to join in at anytime and

Re: [DISCUSS] Integration/e2e test infrastructure requirements

I am going to start the process of creating Jiras out of these initial requirements. I agree with them and think they are a good starting point. Feel free to join in at anytime and add/change/remove requirements as needed. I will update the thread once I have the initial Jiras created and we can

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/859#discussion_r156755792 --- Diff: metron-deployment/amazon-ec2/README.md --- @@ -126,6 +126,10 @@ To provision only subsets of the entire Metron deployment, Ansible tags can

Metron - Emailing Alerts

Hello, Just wondering if Metron has a feature to email alerts based on rules that a user defines. Example: Rule A: Email the user 1...@1.com whenever ip_src_addr=100.2.10.* Rule B: Email the user 1...@1.com whenever payload contains "critical" If not, does anyone have any recommendations on

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/859#discussion_r156751241 --- Diff: metron-deployment/amazon-ec2/README.md --- @@ -126,6 +126,10 @@ To provision only subsets of the entire Metron deployment, Ansible tags can

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/859#discussion_r156748130 --- Diff: metron-deployment/amazon-ec2/README.md --- @@ -126,6 +126,10 @@ To provision only subsets of the entire Metron deployment, Ansible tags can be s

[GitHub] metron pull request #866: METRON-1349 Full Dev Builds Metron Twice

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/866#discussion_r156746647 --- Diff: metron-deployment/playbooks/metron_install.yml --- @@ -15,13 +15,6 @@ # limitations under the License. # --- -- hosts:

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/859#discussion_r156744095 --- Diff: metron-deployment/amazon-ec2/README.md --- @@ -126,6 +126,10 @@ To provision only subsets of the entire Metron deployment, Ansible tags can be s

[GitHub] metron pull request #866: METRON-1349 Full Dev Builds Metron Twice

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/866#discussion_r156737918 --- Diff: metron-deployment/playbooks/metron_install.yml --- @@ -15,13 +15,6 @@ # limitations under the License. # --- -- hosts:

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/859#discussion_r156737230 --- Diff: metron-deployment/amazon-ec2/README.md --- @@ -126,6 +126,10 @@ To provision only subsets of the entire Metron deployment, Ansible tags can

[GitHub] metron pull request #866: METRON-1349 Full Dev Builds Metron Twice

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/866#discussion_r156726426 --- Diff: metron-deployment/playbooks/metron_install.yml --- @@ -15,13 +15,6 @@ # limitations under the License. # --- -- hosts:

[GitHub] metron pull request #866: METRON-1349 Full Dev Builds Metron Twice

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/866#discussion_r156725657 --- Diff: metron-deployment/roles/ambari_config/tasks/main.yml --- @@ -26,16 +26,15 @@ retries: 5 delay: 10 -- name : check if

[GitHub] metron pull request #866: METRON-1349 Full Dev Builds Metron Twice

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/866#discussion_r156724938 --- Diff: metron-deployment/roles/epel/tasks/main.yml --- @@ -16,6 +16,4 @@ # --- - name: Install EPEL repository - yum:

[GitHub] metron pull request #866: METRON-1349 Full Dev Builds Metron Twice

GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/866 METRON-1349 Full Dev Builds Metron Twice Removing the "Quick Dev" environment in #852 had an unintended side effect. It caused Metron to be built twice during the Full Dev deployment process.

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/859#discussion_r156723676 --- Diff: metron-deployment/roles/ambari_config/vars/small_cluster.yml --- @@ -87,6 +87,8 @@ configurations: topology.classpath: '{{

[GitHub] metron issue #862: METRON-1343: Swagger UI for User Controller needs request...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/862 Please take care to mark the jira as done ---

[GitHub] metron pull request #862: METRON-1343: Swagger UI for User Controller needs ...

Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/862 ---

[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/857 Follow up from @merrimanr and my work yesterday. We upped the versions of Node to 9.2.1. Per the doc, >8 is required to work with async/await. For good measure, I also set the NPM version to 5.6.0.

[GitHub] metron issue #863: METRON-1347: Indexing Topology should fail tuples without...

Github user cestella commented on the issue: https://github.com/apache/metron/pull/863 Actually, I don't think `original_string` is required past the parser topology. For instance, profiler messages into enrichment do not have `original_string`. ---

[GitHub] metron issue #862: METRON-1343: Swagger UI for User Controller needs request...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/862 +1, Thanks for the contribution! ---

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/865 METRON-1212 The bundle System and Maven Plugin (Feature Branch) This PR contains the Bundle system and Maven Plugin. The bundle system and the plugin are adapted from the Apache Nifi

[GitHub] metron pull request #774: METRON-1212 The bundle system and maven plugin

Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/774 ---

[GitHub] metron issue #863: METRON-1347: Indexing Topology should fail tuples without...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/863 The minimum required fields, as far as I can see right now are source.type, original_string and timestamp. Given the use case for this is something that has skipped the parser topology, we

[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...

Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r156676868 --- Diff: metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java --- @@ -229,17 +239,30 @@ public

[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r156676155 --- Diff: metron-platform/metron-indexing/README.md --- @@ -15,6 +15,12 @@ Indices are written in batch and the batch size and batch timeout are specified

[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r156675356 --- Diff: metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java --- @@ -229,17 +239,30 @@ public void

[GitHub] metron issue #863: METRON-1347: Indexing Topology should fail tuples without...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/863 I would like to hear feedback from @ottobackwards on other required fields but this looks good to me otherwise. ---

[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...

Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r156674159 --- Diff: metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java --- @@ -229,17 +239,30 @@ public void

Re: [DISCUSS] Community Meetings

I am ok with just notes and no recording. On December 13, 2017 at 04:37:20, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Good points Larry, we would need to get consent from everyone on the call to record to properly comply with regulations in some countries. We would definitely

Re: [DISCUSS] Community Meetings

Good points Larry, we would need to get consent from everyone on the call to record to properly comply with regulations in some countries. We would definitely need someone to step up as note taker. Something else to think about is intended audience. Previously we’ve had meeting like this