[GitHub] metron issue #693: METRON-739 Create Local Profile Runner

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/693
  
Depends if it has dependencies.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #693: METRON-739 Create Local Profile Runner

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/693
  
It is just part of the Profiler Client lib.  Do we need to do anything 
special?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #693: METRON-739 Create Local Profile Runner

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/693
  
@nickwallen , I think this would be great to put in the stellar stand alone 
shell, we should think of how to package it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #708: Metron 1120

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/708

Metron 1120

[METRON-1120](https://issues.apache.org/jira/browse/METRON-1120)

- [ ] This is built on top of METRON-1120 so this should not be committed 
before METRON-1120.

The `groupBy` expression can now reference any of these variables.
* `profile` The name of the profile.
* `entity` The name of the entity being profiled.
* `start` The start time of the profile period in epoch milliseconds.
* `end` The end time of the profile period in epoch milliseconds.
* `duration` The duration of the profile period in milliseconds.
* `result` The result of executing the `result` expression.

Unit tests have been added to validate this functionality. The README has 
also been updated to describe the fields available to the `groupBy` expression.

This can also be tested manually in either a live Profiler or using the 
Profiler debugging functions. 
 The following shows how this change would be used to implement the 
problematic profile described in.

Create a profile that references the start of the profile period in the 
`groupBy` expression.
```
[Stellar]>>> conf := SHELL_EDIT()
[Stellar]>>> conf
{
  "profiles": [
{
  "profile": "calender-effects",
  "onlyif":  "exists(ip_src_addr) and exists(timestamp)",
  "foreach": "ip_src_addr",
  "init":{ "count": 0 },
  "update":  { "count": "count + 1" },
  "result":  "count",
  "groupBy": ["DAY_OF_WEEK(start)"]
}
  ]
}
```

Create a message to exercise the profiler.
```
[Stellar]>>> msg := SHELL_EDIT()
[Stellar]>>> msg
{
"ip_src_addr":"10.0.0.1",
"timestamp":"2017-08-18 09:00:00"
}
```

Create a Profiler and apply the messages to it.
```
[Stellar]>>> p := PROFILER_INIT(conf)
[Stellar]>>> PROFILER_APPLY(msg, p)
org.apache.metron.profiler.StandAloneProfiler@4572b5b4
[Stellar]>>> PROFILER_APPLY(msg, p)
org.apache.metron.profiler.StandAloneProfiler@4572b5b4
[Stellar]>>> PROFILER_APPLY(msg, p)
org.apache.metron.profiler.StandAloneProfiler@4572b5b4
```

Flush the profile and validate the result of executing the `groupBy`.  The 
value is 6, which indicates Friday, which is correct in this case.
```
[Stellar]>>> PROFILER_FLUSH(p)
[{period={duration=90, period=1670094, start=150308460, 
end=150308550}, profile=calender-effects, groups=[6], value=3, 
entity=10.0.0.1}]
```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1120

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/708.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #708


commit 5579748ad4336a7c1a15c319d59fd6cbdeb6531e
Author: Nick Allen 
Date:   2017-08-18T17:37:01Z

METRON-1121 Ignore Profile with Bad 'init', 'update' or 'groupBy'

commit 893b7db84f155ea6af975ee51338f39b763eaedb
Author: Nick Allen 
Date:   2017-08-18T17:45:50Z

Rm errant comment

commit da365c8b546678bbe07011e10ab3cd222faa8297
Author: Nick Allen 
Date:   2017-08-18T19:01:26Z

METRON-1120 Profile's 'groupBy' Expression Has No Reference to Time

commit 5d8a7a06096d5aa725a0ce3b47fef36a8e14ac72
Author: Nick Allen 
Date:   2017-08-18T19:04:35Z

Rm artifacts that should not be in Git

commit c52dce2be6146127eed9af0d2b311ff65f0de551
Author: Nick Allen 
Date:   2017-08-18T19:32:07Z

Updated README

commit 54f1c5969268032e0841f0dd4b5e76449b8b3b6f
Author: Nick Allen 
Date:   2017-08-18T19:35:31Z

Fix README




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [ANNOUNCE] Metron community meeting

[zeo...@gmail.com]

Is it possible to reschedule this to later in the day or another day?  That
overlaps with the eclipse on the east cost of the US that some people would
like to enjoy.

Jon

On Fri, Aug 18, 2017, 13:48 James Sirota  wrote:

> I would like to propose a meeting with the following set of topics:
>
> - What do we want in the next build
> - Where do we want to take the project long-term
> - Any feature requests/comments on existing feature set
> - Feedback on some of the early UI work
>
> Please suggest if there is anything else you want to talk about.
>
> I would like to schedule it for Monday, Aug. 21. at 10.30AM PST.
>
> I would like to have it over webex:
>
>
> https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93
> Meeting number: 622 537 955
> Meeting password: biFTEuh2
>
> For global callers:
>
> https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1
>
>
> Anyone is welcome to join.
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubated and Hatched)
> jsirota AT apache DOT org
>
-- 

Jon

[ANNOUNCE] Metron community meeting

[James Sirota]

I would like to propose a meeting with the following set of topics:

- What do we want in the next build
- Where do we want to take the project long-term
- Any feature requests/comments on existing feature set
- Feedback on some of the early UI work

Please suggest if there is anything else you want to talk about.

I would like to schedule it for Monday, Aug. 21. at 10.30AM PST.

I would like to have it over webex:

https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93
Meeting number: 622 537 955
Meeting password: biFTEuh2

For global callers:
https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1


Anyone is welcome to join. 
--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubated and Hatched)
jsirota AT apache DOT org

[GitHub] metron pull request #526: Metron-846: Add E2E tests for metron management ui

[iraghumitra]

GitHub user iraghumitra reopened a pull request:

https://github.com/apache/metron/pull/526

Metron-846: Add E2E tests for metron management ui

## Contributor Comments
This PR adds e2e test support for management ui. We have few e2e tests in 
management ui but they do not work on quick-dev-platform. The PR makes 
significant changes to the existing e2e code so I would request to review the 
entire e2e tests in total.

- The e2e tests are written to run on quick-dev-platform
- Prerequisites for running e2e test are documented in metron-config README
- Steps to run the e2e tests are documented in metron-config README 
- Implemented promises wherever applicable in .po files


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iraghumitra/incubator-metron METRON-846

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/526.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #526


commit 21300e21c99db32e2c4921f389db3bab6f6271da
Author: rmerriman 
Date:   2016-12-06T00:10:54Z

Merge branch 'middleware' into webapp

commit a7dea157fb9b02739ed9d4f5a647d97b086d4822
Author: rmerriman 
Date:   2016-12-06T00:11:57Z

Resolved licensing issues

commit d38b6265b4c0806826db6f77c7c036f6115db53e
Author: rmerriman 
Date:   2016-12-06T00:12:38Z

Resolved licensing issues

commit f1ce2d0ab740b9c67f4abe00e8e88d81b43dc8cd
Author: rmerriman 
Date:   2016-12-06T00:12:54Z

Merge branch 'docker' into middleware

commit a7b9e7e96aa339e922ec6d2098563f726ced
Author: rmerriman 
Date:   2016-12-06T00:13:32Z

Merge branch 'middleware' into webapp

# Conflicts:
#   pom.xml

commit f7bf12e12c979f541b4ac16174db39d1bf739bae
Author: rmerriman 
Date:   2016-12-06T00:15:28Z

Resolved licensing issues

commit 

Re: [Question] Stopping Storm, Metron & Kafka doesn't stop all Storm processes?

[Laurens Vets]

I'll see what I can do re batch sizes.

I can pretty much gather any metric you want. Which one do you need? 

On 2017-08-18 03:22, zeo...@gmail.com wrote:

> I'm grasping at straws but maybe play with your batch sizes?  I hear now 
> there's a time based option too that I haven't played with. 
> 
> It would be helpful to have more metrics about resource utilization over time 
> in your cluster, are you able to gather anything and maybe put it into a 
> visualization tool? 
> 
> Jon 
> 
> On Thu, Aug 17, 2017, 16:35 Laurens Vets  wrote: 
> 
>> That seems close to the issue that I'm having... But there's a part I
>> don't quite understand in my case.
>> 
>> Everything's working fine for days and then suddenly, Java throws an
>> error (see other mail to the mailing list) and I can't get Metron stable
>> again. It's always either the indexingBolt or hdfsIndexingBolt. The
>> error I see the most after this is "java.lang.OutOfMemoryError: GC
>> overhead limit exceeded"...
>> 
>> It seems that somewhere there's a big pileup of messages which Metron
>> suddenly can't process.
>> 
>> Any ideas on how to further troubleshoot this?
>> 
>> On 2017-08-17 11:10, zeo...@gmail.com wrote:
>>> I used to run into similar issues when my environment was resource
>>> constrained but never ran it to root cause.  It has been a long time
>>> since
>>> I was in this scenario to re-test.
>>> 
>>> https://issues.apache.org/jira/projects/METRON/issues/METRON-485
>>> 
>>> Jon
>>> 
>>> On Thu, Aug 17, 2017 at 12:49 PM Laurens Vets 
>>> wrote:
>>> 
 Hi,
 
 Simple question, when I stop Metron, Kafka & Storm via Ambari, I still
 see the storm worker processes running, is this expected?
 
> 
> -- 
> 
> Jon

Re: [Question] Stopping Storm, Metron & Kafka doesn't stop all Storm processes?

[Laurens Vets]

I'll see what I can do re batch sizes.

I can pretty much gather any metric you want. Which one do you need?

On 2017-08-18 03:22, zeo...@gmail.com wrote:
I'm grasping at straws but maybe play with your batch sizes?  I hear 
now

there's a time based option too that I haven't played with.

It would be helpful to have more metrics about resource utilization 
over
time in your cluster, are you able to gather anything and maybe put it 
into

a visualization tool?

Jon

On Thu, Aug 17, 2017, 16:35 Laurens Vets  wrote:


That seems close to the issue that I'm having... But there's a part I
don't quite understand in my case.

Everything's working fine for days and then suddenly, Java throws an
error (see other mail to the mailing list) and I can't get Metron 
stable

again. It's always either the indexingBolt or hdfsIndexingBolt. The
error I see the most after this is "java.lang.OutOfMemoryError: GC
overhead limit exceeded"...

It seems that somewhere there's a big pileup of messages which Metron
suddenly can't process.

Any ideas on how to further troubleshoot this?

On 2017-08-17 11:10, zeo...@gmail.com wrote:
> I used to run into similar issues when my environment was resource
> constrained but never ran it to root cause.  It has been a long time
> since
> I was in this scenario to re-test.
>
> https://issues.apache.org/jira/projects/METRON/issues/METRON-485
>
> Jon
>
> On Thu, Aug 17, 2017 at 12:49 PM Laurens Vets 
> wrote:
>
>> Hi,
>>
>> Simple question, when I stop Metron, Kafka & Storm via Ambari, I still
>> see the storm worker processes running, is this expected?
>>

[GitHub] metron pull request #699: METRON-1068: Use Elastic end point in alerts ui

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/699#discussion_r133964660
  
--- Diff: metron-interface/metron-alerts/e2e/login/login.e2e-spec.ts ---
@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { LoginPage } from './login.po';
+
+describe('login to application', function() {
+let page: LoginPage;
+
+beforeEach(() => {
+page = new LoginPage();
+});
+
+it('should display error message for invalid credentials', () => {
--- End diff --

Ok, that's what I thought.  There is a dependency on Full Dev to run the 
E2E tests.

Do we have to do this?  Does it make more sense to mock the REST API, so 
the E2E tests can be run without this dependency?  It seems to be that it is 
going to be far easier to fully test border conditions and such if the E2E 
tests run against a mocked REST API.  Or is that crazy talk?  

But assuming we leave it as-is, then we should update the steps in the 
README for running the E2E test suite.  We should have an explicit step to 
start Full Dev.





---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #703: METRON-1115: Jackson is not properly serializing readMeta...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/703
  
Also updated the REST SensorParserConfig test to catch this type of error 
in the future.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #699: METRON-1068: Use Elastic end point in alerts ui

[iraghumitra]

Github user iraghumitra commented on a diff in the pull request:

https://github.com/apache/metron/pull/699#discussion_r133943020
  
--- Diff: metron-interface/metron-alerts/e2e/login/login.e2e-spec.ts ---
@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { LoginPage } from './login.po';
+
+describe('login to application', function() {
+let page: LoginPage;
+
+beforeEach(() => {
+page = new LoginPage();
+});
+
+it('should display error message for invalid credentials', () => {
--- End diff --

Yes, I need to connect to a rest-api source for authentication. I hard 
coded it to node1:8082 since quick-dev and full-dev create's boxes with this 
name. I can make it configurable if you see an issue with this. I thought it 
would just make things simple.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Question] Stopping Storm, Metron & Kafka doesn't stop all Storm processes?

[zeo...@gmail.com]

I'm grasping at straws but maybe play with your batch sizes?  I hear now
there's a time based option too that I haven't played with.

It would be helpful to have more metrics about resource utilization over
time in your cluster, are you able to gather anything and maybe put it into
a visualization tool?

Jon

On Thu, Aug 17, 2017, 16:35 Laurens Vets  wrote:

> That seems close to the issue that I'm having... But there's a part I
> don't quite understand in my case.
>
> Everything's working fine for days and then suddenly, Java throws an
> error (see other mail to the mailing list) and I can't get Metron stable
> again. It's always either the indexingBolt or hdfsIndexingBolt. The
> error I see the most after this is "java.lang.OutOfMemoryError: GC
> overhead limit exceeded"...
>
> It seems that somewhere there's a big pileup of messages which Metron
> suddenly can't process.
>
> Any ideas on how to further troubleshoot this?
>
> On 2017-08-17 11:10, zeo...@gmail.com wrote:
> > I used to run into similar issues when my environment was resource
> > constrained but never ran it to root cause.  It has been a long time
> > since
> > I was in this scenario to re-test.
> >
> > https://issues.apache.org/jira/projects/METRON/issues/METRON-485
> >
> > Jon
> >
> > On Thu, Aug 17, 2017 at 12:49 PM Laurens Vets 
> > wrote:
> >
> >> Hi,
> >>
> >> Simple question, when I stop Metron, Kafka & Storm via Ambari, I still
> >> see the storm worker processes running, is this expected?
> >>
>
-- 

Jon