[GitHub] metron issue #693: METRON-739 Create Local Profile Runner
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/693 Depends if it has dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #693: METRON-739 Create Local Profile Runner
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/693 It is just part of the Profiler Client lib. Do we need to do anything special? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #693: METRON-739 Create Local Profile Runner
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/693 @nickwallen , I think this would be great to put in the stellar stand alone shell, we should think of how to package it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #708: Metron 1120
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/708 Metron 1120 [METRON-1120](https://issues.apache.org/jira/browse/METRON-1120) - [ ] This is built on top of METRON-1120 so this should not be committed before METRON-1120. The `groupBy` expression can now reference any of these variables. * `profile` The name of the profile. * `entity` The name of the entity being profiled. * `start` The start time of the profile period in epoch milliseconds. * `end` The end time of the profile period in epoch milliseconds. * `duration` The duration of the profile period in milliseconds. * `result` The result of executing the `result` expression. Unit tests have been added to validate this functionality. The README has also been updated to describe the fields available to the `groupBy` expression. This can also be tested manually in either a live Profiler or using the Profiler debugging functions. The following shows how this change would be used to implement the problematic profile described in. Create a profile that references the start of the profile period in the `groupBy` expression. ``` [Stellar]>>> conf := SHELL_EDIT() [Stellar]>>> conf { "profiles": [ { "profile": "calender-effects", "onlyif": "exists(ip_src_addr) and exists(timestamp)", "foreach": "ip_src_addr", "init":{ "count": 0 }, "update": { "count": "count + 1" }, "result": "count", "groupBy": ["DAY_OF_WEEK(start)"] } ] } ``` Create a message to exercise the profiler. ``` [Stellar]>>> msg := SHELL_EDIT() [Stellar]>>> msg { "ip_src_addr":"10.0.0.1", "timestamp":"2017-08-18 09:00:00" } ``` Create a Profiler and apply the messages to it. ``` [Stellar]>>> p := PROFILER_INIT(conf) [Stellar]>>> PROFILER_APPLY(msg, p) org.apache.metron.profiler.StandAloneProfiler@4572b5b4 [Stellar]>>> PROFILER_APPLY(msg, p) org.apache.metron.profiler.StandAloneProfiler@4572b5b4 [Stellar]>>> PROFILER_APPLY(msg, p) org.apache.metron.profiler.StandAloneProfiler@4572b5b4 ``` Flush the profile and validate the result of executing the `groupBy`. The value is 6, which indicates Friday, which is correct in this case. ``` [Stellar]>>> PROFILER_FLUSH(p) [{period={duration=90, period=1670094, start=150308460, end=150308550}, profile=calender-effects, groups=[6], value=3, entity=10.0.0.1}] ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1120 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/708.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #708 commit 5579748ad4336a7c1a15c319d59fd6cbdeb6531e Author: Nick Allen Date: 2017-08-18T17:37:01Z METRON-1121 Ignore Profile with Bad 'init', 'update' or 'groupBy' commit 893b7db84f155ea6af975ee51338f39b763eaedb Author: Nick Allen Date: 2017-08-18T17:45:50Z Rm errant comment commit da365c8b546678bbe07011e10ab3cd222faa8297 Author: Nick Allen Date: 2017-08-18T19:01:26Z METRON-1120 Profile's 'groupBy' Expression Has No Reference to Time commit 5d8a7a06096d5aa725a0ce3b47fef36a8e14ac72 Author: Nick Allen Date: 2017-08-18T19:04:35Z Rm artifacts that should not be in Git commit c52dce2be6146127eed9af0d2b311ff65f0de551 Author: Nick Allen Date: 2017-08-18T19:32:07Z Updated README commit 54f1c5969268032e0841f0dd4b5e76449b8b3b6f Author: Nick Allen Date: 2017-08-18T19:35:31Z Fix README --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [ANNOUNCE] Metron community meeting
Is it possible to reschedule this to later in the day or another day? That overlaps with the eclipse on the east cost of the US that some people would like to enjoy. Jon On Fri, Aug 18, 2017, 13:48 James Sirota wrote: > I would like to propose a meeting with the following set of topics: > > - What do we want in the next build > - Where do we want to take the project long-term > - Any feature requests/comments on existing feature set > - Feedback on some of the early UI work > > Please suggest if there is anything else you want to talk about. > > I would like to schedule it for Monday, Aug. 21. at 10.30AM PST. > > I would like to have it over webex: > > > https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93 > Meeting number: 622 537 955 > Meeting password: biFTEuh2 > > For global callers: > > https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC&ED=590161912&tollFree=1 > > > Anyone is welcome to join. > --- > Thank you, > > James Sirota > PPMC- Apache Metron (Incubated and Hatched) > jsirota AT apache DOT org > -- Jon
[ANNOUNCE] Metron community meeting
I would like to propose a meeting with the following set of topics: - What do we want in the next build - Where do we want to take the project long-term - Any feature requests/comments on existing feature set - Feedback on some of the early UI work Please suggest if there is anything else you want to talk about. I would like to schedule it for Monday, Aug. 21. at 10.30AM PST. I would like to have it over webex: https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93 Meeting number: 622 537 955 Meeting password: biFTEuh2 For global callers: https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC&ED=590161912&tollFree=1 Anyone is welcome to join. --- Thank you, James Sirota PPMC- Apache Metron (Incubated and Hatched) jsirota AT apache DOT org
[GitHub] metron pull request #707: METRON-1121 Ignore Profile with Bad 'init', 'updat...
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/707 METRON-1121 Ignore Profile with Bad 'init', 'update' or 'groupBy' The Profiler needs to gracefully ignore profiles that have a bad 'init', 'update', or 'groupBy' expression. Under certain conditions a message may not be applied to all valid profiles, due to a single bad profile. If the Profiler is running with multiple profiles defined and one of those profiles has a Stellar expression that throws an exception for either the 'init', 'update', or 'groupBy', the current logic will not continue to attempt to apply the message to the remaining valid profiles. Only valid profiles executed before the bad profile will have the message applied. ## Testing Unit tests have been added to check for these conditions. This can be manually tested with either a live Profiler running in a cluster or using the Profiler debugging functions. Create two profiles; one that is valid and another that is invalid, like the following. Ensure that all test messages that are sent through the Profiler, reach the `good-profile` despite the `bad-profile`. ``` { "profiles": [ { "profile": "bad-profile", "foreach": "ip_src_addr", "init": { "x": "2 / 0" }, "update": { "x": "x + 1" }, "result": "x" }, { "profile": "good-profile", "foreach": "ip_src_addr", "init": { "x": "0" }, "update": { "x": "x + 1" }, "result": "x" } ] } ``` ## Pull Request Checklist - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1121 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/707.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #707 commit 5579748ad4336a7c1a15c319d59fd6cbdeb6531e Author: Nick Allen Date: 2017-08-18T17:37:01Z METRON-1121 Ignore Profile with Bad 'init', 'update' or 'groupBy' commit 893b7db84f155ea6af975ee51338f39b763eaedb Author: Nick Allen Date: 2017-08-18T17:45:50Z Rm errant comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #526: Metron-846: Add E2E tests for metron management ui
Github user iraghumitra closed the pull request at: https://github.com/apache/metron/pull/526 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #526: Metron-846: Add E2E tests for metron management ui
GitHub user iraghumitra reopened a pull request: https://github.com/apache/metron/pull/526 Metron-846: Add E2E tests for metron management ui ## Contributor Comments This PR adds e2e test support for management ui. We have few e2e tests in management ui but they do not work on quick-dev-platform. The PR makes significant changes to the existing e2e code so I would request to review the entire e2e tests in total. - The e2e tests are written to run on quick-dev-platform - Prerequisites for running e2e test are documented in metron-config README - Steps to run the e2e tests are documented in metron-config README - Implemented promises wherever applicable in .po files ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron (Incubating). Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book bin/generate-md.sh mvn site:site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/iraghumitra/incubator-metron METRON-846 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/526.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #526 commit 21300e21c99db32e2c4921f389db3bab6f6271da Author: rmerriman Date: 2016-12-06T00:10:54Z Merge branch 'middleware' into webapp commit a7dea157fb9b02739ed9d4f5a647d97b086d4822 Author: rmerriman Date: 2016-12-06T00:11:57Z Resolved licensing issues commit d38b6265b4c0806826db6f77c7c036f6115db53e Author: rmerriman Date: 2016-12-06T00:12:38Z Resolved licensing issues commit f1ce2d0ab740b9c67f4abe00e8e88d81b43dc8cd Author: rmerriman Date: 2016-12-06T00:12:54Z Merge branch 'docker' into middleware commit a7b9e7e96aa339e922ec6d2098563f726ced Author: rmerriman Date: 2016-12-06T00:13:32Z Merge branch 'middleware' into webapp # Conflicts: # pom.xml commit f7bf12e12c979f541b4ac16174db39d1bf739bae Author: rmerriman Date: 2016-12-06T00:15:28Z Resolved licensing issues commit 25bc44e3f50f21b447d0217a2eccb4184f743c24 Author: rmerriman Date: 2016-12-06T17:10:37Z Updated scripts to add Hibernate and MySQL client to classpath commit c03e83c012b6a9c007a3d584ce1a7a99e6a2
Re: [Question] Stopping Storm, Metron & Kafka doesn't stop all Storm processes?
I'll see what I can do re batch sizes. I can pretty much gather any metric you want. Which one do you need? On 2017-08-18 03:22, zeo...@gmail.com wrote: > I'm grasping at straws but maybe play with your batch sizes? I hear now > there's a time based option too that I haven't played with. > > It would be helpful to have more metrics about resource utilization over time > in your cluster, are you able to gather anything and maybe put it into a > visualization tool? > > Jon > > On Thu, Aug 17, 2017, 16:35 Laurens Vets wrote: > >> That seems close to the issue that I'm having... But there's a part I >> don't quite understand in my case. >> >> Everything's working fine for days and then suddenly, Java throws an >> error (see other mail to the mailing list) and I can't get Metron stable >> again. It's always either the indexingBolt or hdfsIndexingBolt. The >> error I see the most after this is "java.lang.OutOfMemoryError: GC >> overhead limit exceeded"... >> >> It seems that somewhere there's a big pileup of messages which Metron >> suddenly can't process. >> >> Any ideas on how to further troubleshoot this? >> >> On 2017-08-17 11:10, zeo...@gmail.com wrote: >>> I used to run into similar issues when my environment was resource >>> constrained but never ran it to root cause. It has been a long time >>> since >>> I was in this scenario to re-test. >>> >>> https://issues.apache.org/jira/projects/METRON/issues/METRON-485 >>> >>> Jon >>> >>> On Thu, Aug 17, 2017 at 12:49 PM Laurens Vets >>> wrote: >>> Hi, Simple question, when I stop Metron, Kafka & Storm via Ambari, I still see the storm worker processes running, is this expected? > > -- > > Jon
Re: [Question] Stopping Storm, Metron & Kafka doesn't stop all Storm processes?
I'll see what I can do re batch sizes. I can pretty much gather any metric you want. Which one do you need? On 2017-08-18 03:22, zeo...@gmail.com wrote: I'm grasping at straws but maybe play with your batch sizes? I hear now there's a time based option too that I haven't played with. It would be helpful to have more metrics about resource utilization over time in your cluster, are you able to gather anything and maybe put it into a visualization tool? Jon On Thu, Aug 17, 2017, 16:35 Laurens Vets wrote: That seems close to the issue that I'm having... But there's a part I don't quite understand in my case. Everything's working fine for days and then suddenly, Java throws an error (see other mail to the mailing list) and I can't get Metron stable again. It's always either the indexingBolt or hdfsIndexingBolt. The error I see the most after this is "java.lang.OutOfMemoryError: GC overhead limit exceeded"... It seems that somewhere there's a big pileup of messages which Metron suddenly can't process. Any ideas on how to further troubleshoot this? On 2017-08-17 11:10, zeo...@gmail.com wrote: > I used to run into similar issues when my environment was resource > constrained but never ran it to root cause. It has been a long time > since > I was in this scenario to re-test. > > https://issues.apache.org/jira/projects/METRON/issues/METRON-485 > > Jon > > On Thu, Aug 17, 2017 at 12:49 PM Laurens Vets > wrote: > >> Hi, >> >> Simple question, when I stop Metron, Kafka & Storm via Ambari, I still >> see the storm worker processes running, is this expected? >>
[GitHub] metron pull request #699: METRON-1068: Use Elastic end point in alerts ui
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/699#discussion_r133964660 --- Diff: metron-interface/metron-alerts/e2e/login/login.e2e-spec.ts --- @@ -0,0 +1,44 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import { LoginPage } from './login.po'; + +describe('login to application', function() { +let page: LoginPage; + +beforeEach(() => { +page = new LoginPage(); +}); + +it('should display error message for invalid credentials', () => { --- End diff -- Ok, that's what I thought. There is a dependency on Full Dev to run the E2E tests. Do we have to do this? Does it make more sense to mock the REST API, so the E2E tests can be run without this dependency? It seems to be that it is going to be far easier to fully test border conditions and such if the E2E tests run against a mocked REST API. Or is that crazy talk? But assuming we leave it as-is, then we should update the steps in the README for running the E2E test suite. We should have an explicit step to start Full Dev. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #703: METRON-1115: Jackson is not properly serializing readMeta...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/703 Also updated the REST SensorParserConfig test to catch this type of error in the future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #699: METRON-1068: Use Elastic end point in alerts ui
Github user iraghumitra commented on a diff in the pull request: https://github.com/apache/metron/pull/699#discussion_r133943020 --- Diff: metron-interface/metron-alerts/e2e/login/login.e2e-spec.ts --- @@ -0,0 +1,44 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import { LoginPage } from './login.po'; + +describe('login to application', function() { +let page: LoginPage; + +beforeEach(() => { +page = new LoginPage(); +}); + +it('should display error message for invalid credentials', () => { --- End diff -- Yes, I need to connect to a rest-api source for authentication. I hard coded it to node1:8082 since quick-dev and full-dev create's boxes with this name. I can make it configurable if you see an issue with this. I thought it would just make things simple. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [Question] Stopping Storm, Metron & Kafka doesn't stop all Storm processes?
I'm grasping at straws but maybe play with your batch sizes? I hear now there's a time based option too that I haven't played with. It would be helpful to have more metrics about resource utilization over time in your cluster, are you able to gather anything and maybe put it into a visualization tool? Jon On Thu, Aug 17, 2017, 16:35 Laurens Vets wrote: > That seems close to the issue that I'm having... But there's a part I > don't quite understand in my case. > > Everything's working fine for days and then suddenly, Java throws an > error (see other mail to the mailing list) and I can't get Metron stable > again. It's always either the indexingBolt or hdfsIndexingBolt. The > error I see the most after this is "java.lang.OutOfMemoryError: GC > overhead limit exceeded"... > > It seems that somewhere there's a big pileup of messages which Metron > suddenly can't process. > > Any ideas on how to further troubleshoot this? > > On 2017-08-17 11:10, zeo...@gmail.com wrote: > > I used to run into similar issues when my environment was resource > > constrained but never ran it to root cause. It has been a long time > > since > > I was in this scenario to re-test. > > > > https://issues.apache.org/jira/projects/METRON/issues/METRON-485 > > > > Jon > > > > On Thu, Aug 17, 2017 at 12:49 PM Laurens Vets > > wrote: > > > >> Hi, > >> > >> Simple question, when I stop Metron, Kafka & Storm via Ambari, I still > >> see the storm worker processes running, is this expected? > >> > -- Jon