Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST

[zeo...@gmail.com]

I won't be able to make it and would really like to make sure there's a
recording for this one, if possible.  I'm unavailable until Thursday of
next week, but not necessarily suggesting this gets moved.

Jon

On Thu, Sep 21, 2017, 15:04 Otto Fowler  wrote:

> I can’t make that time, can we make it later in the day?
>
>
> On September 21, 2017 at 11:40:37, James Sirota (jsir...@apache.org)
> wrote:
>
> https://hortonworks.webex.com/meet/jsirota
>
-- 

Jon

[GitHub] metron issue #762: METRON-1189: Add alert escalation to the Alerts UI

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/762
  
I researched the saveRefreshState() and restoreRefreshState() addition and 
it uncovered a bug which then uncovered a problem with how our UI works.

The bug was that the selectedAlerts field (checkboxes in the list) was not 
getting cleared on any of the bulk actions.  As refreshes happened the 
checkboxes would disappear and be replaced by new results but that that field 
would constantly grow because they were never unchecked on the previous page. 
It was benign before this PR but is now causing a lot of duplicate REST 
requests.  We need to figure out the right way to clear these when a bulk 
action finishes but there is also another issue.

As I was testing for this I realized when I set the refresh rate to a high 
rate I had to click quickly or the page would change and I would lose my 
selections.  What @iraghumitra mentions above is a pause while the REST 
requests are being processed but I think it makes sense to pause even earlier, 
whenever you have a checkbox checked.  I can obviously get it to work if I 
pause it but I think that's a better UX.


---

Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST

[Otto Fowler]

I can’t make that time, can we make it later in the day?


On September 21, 2017 at 11:40:37, James Sirota (jsir...@apache.org) wrote:

https://hortonworks.webex.com/meet/jsirota

Re: feature branch bumps

[Otto Fowler]

My thought was that in answering things in the wiki, it would build out the
‘guide’ there.  But I was just taking a stab at that.  I am open to
whatever the group thinks would work best.



On September 21, 2017 at 14:47:10, Nick Allen (n...@nickallen.org) wrote:

> What I would like to do, and what it seems to me that we need to do (
again I apologize if I am wrong ), is to catch up on the confluence and
where things currently stand, and then move the discussion on from there.

Perfect, thanks.  I think you are completely right.  I will review that in
detail.

Should I respond with questions in the comments of the Wiki or some other
way?  Whatever works for you.


On Thu, Sep 21, 2017 at 2:24 PM Otto Fowler  wrote:

> Nick, thanks for taking the time to reply, I have no doubts about your
> motivations and intent.  Hopefully we get through this point by point
> stuff, which is always difficult on email.
>
>
> https://cwiki.apache.org/confluence/display/METRON/Metron+Extension+System+and+Parser+Extensions
>
> This is my recent attempt at creating a framework for the review.  It
> includes testing areas, review areas etc.  I have not received any feedback
> on it so I have to assume that
> you have not seen it.  My stated hope is that through questions about the
> confluence I can address anything missing to make things reviewable, or
> more reviewable. If you have reviewed it, and are just not referring to it,
> I apologize.
> Please take a look, and provide feedback so that I can address it there if
> you have not.
>
> As far as the problem being lack of community commitment, that is not what
> I was trying to imply.  I was, in my mind agreeing with you that the size
> of the changes has proven to be a barrier in practice, despite our
> agreement in 777 to proceed as is, and our discussion about the feature
> branch.
>
> As far as (1) the scope, I tried to lay out the areas of change and what
> was changed in the description of 777.  I also re-listed separately the
> specific files changed, and what functional area the change referred to in
> the PR.  Can you point me to an example of a different definition of scope
> that I can copy?  Is it a more descriptive but still high level statement
> that would help?
>
> As far as (2) I have outlined the parts that could be considered
> separately ( bundles, archetype, maven plugin ).  Once we switch to the new
> parser structure, there is no way to cut it down and still have a working
> build, because the new parser structure and loading touches pretty much all
> levels.  This is not that different from what we discussed at the time.  I
> do not know what is in there that should not be in there, aside from the 3
> areas I mentioned that could be reviewed as not integrated components.  Are
> you referring to those areas or something else?
>
> As far as (3) - In the original 777, there was no user facing changes.
> Only developer changes, which I addressed in the adding system parsers
> guide based on Mike’s feedback.  Maybe we are using different terminology
> and just missing each other.  Based on the original 777, a user ( someone
> who deploys metron and uses the ambari configuration and the management ui,
> pushes new parsers to zookeeper, uses kibana etc) would not see any
> changes, or should not have seen any changes. There was no new
> functionality to test from a user POV.  Mike did find a regression in the
> config ui wrt grok rules editing, which I addressed, but that was not new
> function problem, but a regression.  The user facing changes where in 942 (
> rest  + stellar management command) and where documented, and are also now
> in the UI pr.  Asking how a user creates a parser in 777 when I
> specifically put that in 942 to cut down on scope, while also questioning
> the scope is confusing to me.
>
> As far as (4) - please see the confluence pages.
>
> What I would like to do, and what it seems to me that we need to do (
> again I apologize if I am wrong ), is to catch up on the confluence and
> where things currently stand, and then move the discussion on from there.
> I feel like a lot has gone on in 777 and there that relate to your concerns
> ( although I am not saying they address them ).
>
>
> On September 21, 2017 at 11:58:51, Nick Allen (n...@nickallen.org) wrote:
>
> But, as you said, it has been since April, and only Bundles has been
>> reviewed. If nobody but Matt is even going to _attempt_ code review, then
>> it may now be implicitly required that I do this.
>
>
> I disagree
> ​ that the problem is lack of community commitment to the change​
> .  We have had quite a few core team members who at least began a review.  I
> can't speak for anyone else, but I'll speak for my own experience
> ​ in attempting to review 777.​
>
> ​These are the reasons why I was not able to close out my review of 777
> with a +1.
>
> ​(1)​
>
> T
> he scope of this change is largely undefined. ​​
> ​I do​
> not
> ​ ​
> ​have ​
> a 

Re: feature branch bumps

[Nick Allen]

> What I would like to do, and what it seems to me that we need to do (
again I apologize if I am wrong ), is to catch up on the confluence and
where things currently stand, and then move the discussion on from there.

Perfect, thanks.  I think you are completely right.  I will review that in
detail.

Should I respond with questions in the comments of the Wiki or some other
way?  Whatever works for you.


On Thu, Sep 21, 2017 at 2:24 PM Otto Fowler  wrote:

> Nick, thanks for taking the time to reply, I have no doubts about your
> motivations and intent.  Hopefully we get through this point by point
> stuff, which is always difficult on email.
>
>
> https://cwiki.apache.org/confluence/display/METRON/Metron+Extension+System+and+Parser+Extensions
>
> This is my recent attempt at creating a framework for the review.  It
> includes testing areas, review areas etc.  I have not received any feedback
> on it so I have to assume that
> you have not seen it.  My stated hope is that through questions about the
> confluence I can address anything missing to make things reviewable, or
> more reviewable. If you have reviewed it, and are just not referring to it,
> I apologize.
> Please take a look, and provide feedback so that I can address it there if
> you have not.
>
> As far as the problem being lack of community commitment, that is not what
> I was trying to imply.  I was, in my mind agreeing with you that the size
> of the changes has proven to be a barrier in practice, despite our
> agreement in 777 to proceed as is, and our discussion about the feature
> branch.
>
> As far as (1) the scope, I tried to lay out the areas of change and what
> was changed in the description of 777.  I also re-listed separately the
> specific files changed, and what functional area the change referred to in
> the PR.  Can you point me to an example of a different definition of scope
> that I can copy?  Is it a more descriptive but still high level statement
> that would help?
>
> As far as (2) I have outlined the parts that could be considered
> separately ( bundles, archetype, maven plugin ).  Once we switch to the new
> parser structure, there is no way to cut it down and still have a working
> build, because the new parser structure and loading touches pretty much all
> levels.  This is not that different from what we discussed at the time.  I
> do not know what is in there that should not be in there, aside from the 3
> areas I mentioned that could be reviewed as not integrated components.  Are
> you referring to those areas or something else?
>
> As far as (3) - In the original 777, there was no user facing changes.
> Only developer changes, which I addressed in the adding system parsers
> guide based on Mike’s feedback.  Maybe we are using different terminology
> and just missing each other.  Based on the original 777, a user ( someone
> who deploys metron and uses the ambari configuration and the management ui,
> pushes new parsers to zookeeper, uses kibana etc) would not see any
> changes, or should not have seen any changes. There was no new
> functionality to test from a user POV.  Mike did find a regression in the
> config ui wrt grok rules editing, which I addressed, but that was not new
> function problem, but a regression.  The user facing changes where in 942 (
> rest  + stellar management command) and where documented, and are also now
> in the UI pr.  Asking how a user creates a parser in 777 when I
> specifically put that in 942 to cut down on scope, while also questioning
> the scope is confusing to me.
>
> As far as (4) - please see the confluence pages.
>
> What I would like to do, and what it seems to me that we need to do (
> again I apologize if I am wrong ), is to catch up on the confluence and
> where things currently stand, and then move the discussion on from there.
> I feel like a lot has gone on in 777 and there that relate to your concerns
> ( although I am not saying they address them ).
>
>
> On September 21, 2017 at 11:58:51, Nick Allen (n...@nickallen.org) wrote:
>
> But, as you said, it has been since April, and only Bundles has been
>> reviewed. If nobody but Matt is even going to _attempt_ code review, then
>> it may now be implicitly required that I do this.
>
>
> I disagree
> ​ that the problem is lack of community commitment to the change​
> .  We have had quite a few core team members who at least began a review.  I
> can't speak for anyone else, but I'll speak for my own experience
> ​ in attempting to review 777.​
>
> ​These are the reasons why I was not able to close out my review of 777
> with a +1.
>
> ​(1)​
>
> T
> he scope of this change is largely undefined. ​​
> ​I do​
> not
> ​ ​
> ​have ​
> a good understanding
> ​ ​
> of what this thing actually does
>
> ​ and does not do
> .  ​What does it touch and what does it not touch?​
>
>
> ​(2) ​
> I feel like there are lots of different pieces of functionality
> ​ ​
> ​in the FB ​
> that we've tied all together 

Re: feature branch bumps

[Otto Fowler]

Nick, thanks for taking the time to reply, I have no doubts about your
motivations and intent.  Hopefully we get through this point by point
stuff, which is always difficult on email.

https://cwiki.apache.org/confluence/display/METRON/Metron+Extension+System+and+Parser+Extensions

This is my recent attempt at creating a framework for the review.  It
includes testing areas, review areas etc.  I have not received any feedback
on it so I have to assume that
you have not seen it.  My stated hope is that through questions about the
confluence I can address anything missing to make things reviewable, or
more reviewable. If you have reviewed it, and are just not referring to it,
I apologize.
Please take a look, and provide feedback so that I can address it there if
you have not.

As far as the problem being lack of community commitment, that is not what
I was trying to imply.  I was, in my mind agreeing with you that the size
of the changes has proven to be a barrier in practice, despite our
agreement in 777 to proceed as is, and our discussion about the feature
branch.

As far as (1) the scope, I tried to lay out the areas of change and what
was changed in the description of 777.  I also re-listed separately the
specific files changed, and what functional area the change referred to in
the PR.  Can you point me to an example of a different definition of scope
that I can copy?  Is it a more descriptive but still high level statement
that would help?

As far as (2) I have outlined the parts that could be considered separately
( bundles, archetype, maven plugin ).  Once we switch to the new parser
structure, there is no way to cut it down and still have a working build,
because the new parser structure and loading touches pretty much all
levels.  This is not that different from what we discussed at the time.  I
do not know what is in there that should not be in there, aside from the 3
areas I mentioned that could be reviewed as not integrated components.  Are
you referring to those areas or something else?

As far as (3) - In the original 777, there was no user facing changes.
Only developer changes, which I addressed in the adding system parsers
guide based on Mike’s feedback.  Maybe we are using different terminology
and just missing each other.  Based on the original 777, a user ( someone
who deploys metron and uses the ambari configuration and the management ui,
pushes new parsers to zookeeper, uses kibana etc) would not see any
changes, or should not have seen any changes. There was no new
functionality to test from a user POV.  Mike did find a regression in the
config ui wrt grok rules editing, which I addressed, but that was not new
function problem, but a regression.  The user facing changes where in 942 (
rest  + stellar management command) and where documented, and are also now
in the UI pr.  Asking how a user creates a parser in 777 when I
specifically put that in 942 to cut down on scope, while also questioning
the scope is confusing to me.

As far as (4) - please see the confluence pages.

What I would like to do, and what it seems to me that we need to do ( again
I apologize if I am wrong ), is to catch up on the confluence and where
things currently stand, and then move the discussion on from there.  I feel
like a lot has gone on in 777 and there that relate to your concerns (
although I am not saying they address them ).


On September 21, 2017 at 11:58:51, Nick Allen (n...@nickallen.org) wrote:

But, as you said, it has been since April, and only Bundles has been
> reviewed. If nobody but Matt is even going to _attempt_ code review, then
> it may now be implicitly required that I do this.


I disagree
​ that the problem is lack of community commitment to the change​
.  We have had quite a few core team members who at least began a review.  I
can't speak for anyone else, but I'll speak for my own experience
​ in attempting to review 777.​

​These are the reasons why I was not able to close out my review of 777
with a +1.

​(1)​

T
he scope of this change is largely undefined. ​​
​I do​
not
​ ​
​have ​
a good understanding
​ ​
of what this thing actually does

​ and does not do
.  ​What does it touch and what does it not touch?​


​(2) ​
I feel like there are lots of different pieces of functionality
​ ​
​in the FB ​
that we've tied all together that doesn't necessarily need to be.
​  ​
With this being one big​
​ ​
chunk of code, its take it all or nothing.


​(3)​
 I asked for documentation on how a user is supposed to interact with this
change.  This should help define what a parser is, how a user creates a
parser, does an analyst have to run Maven just to deploy a parser?  ​

​

​(4)​
 I asked for a complete test plan on how to test all of this functionality.



​I don't think I received for very good responses from you on these
questions.  Comments from you like this [1], do not help me review the code.

Since this PR is not adding new functionality the testing is the usual
> smoke test for 

[GitHub] metron pull request #756: METRON-1182: Refactored AlertList View to abstract...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/756


---

[GitHub] metron issue #756: METRON-1182: Refactored AlertList View to abstract displa...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/756
  
Lots of improvements in this PR, I like it.  Spun this up in full dev and 
everything continues to work.  Nice job.  +1


---

[GitHub] metron issue #757: METRON-938: "service metron-rest start " does n...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/757
  
Spun it up again in full dev, setup MySQL and everything works as it 
should.  Thanks so much @justinleet for fixing this!  Turned out great.  +1


---

[DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST

[James Sirota]

Hi Guys,

I'd like to propose a community meeting for this Tuesday, Sept.23 and focus on 
the strategy for getting Metron-777 and related PRs on Otto's feature branch 
merged into master.  Otto, is there a way you can record a youtube of adding a 
custom via your feature branch by Tueday? I'd like to be able to review that 
and get everyone's feedback so that we can start formulating our strategy on 
getting it in.

I propose to have this meeting at  Tuesday, Sept.23 10AM PST if that works for 
everyone:

https://hortonworks.webex.com/meet/jsirota
Call-in toll-free number (US/Canada) 1-877-668-4493



--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

[GitHub] metron issue #737: METRON-1161: Add ability to edit parser command line opti...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/737
  
Changing numWorkers and numAckers default values from null to 1 had the 
unintended consequences of breaking the ParserTopologyCliTest.  This is because 
they can no longer be overriden by setting the topology.workers and 
topology.acker.executors properties in stormConfig.  I believe this is the 
correct behaviour because numWorkers and numAckers are top-level properties and 
should take precedence.  If this is not correct then I would argue that we 
shouldn't even have them.  Can you verify this @cestella?


---

[GitHub] metron pull request #769: METRON-1198: Pycapa - No such configuration proper...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/769

METRON-1198: Pycapa - No such configuration property: 
"sasl.kerberos.principal"

When running Pycapa in a Kerberized environment, but without a version of 
Librdkafka built with SASL support, it can produce error messages that 
look-like the following.

```
KafkaError{code=_INVALID_ARG,val=-186,str="No such configuration property: 
"sasl.kerberos.principal""}
```

This can happen when a user accidentally installs multiple version of 
Librdkafka and the version that the Python interpreter links to is the one 
without SASL support.  I am going to update the README to doc this specific 
condition.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron patch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/769.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #769


commit beaeca920ebb2de7131fdb2e0e036f016874d530
Author: Nick Allen 
Date:   2017-09-21T13:39:18Z

METRON-1198: Pycapa - No such configuration property: 
"sasl.kerberos.principal"




---

[GitHub] metron issue #760: METRON-1188: Ambari global configuration management broke...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/760
  
Thanks for making the adjustments, @mmiklavc.   I'm +1 on this.  It's a 
great step forward and gives a good foundation for future improvements we want 
to make.


---

[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...

[iraghumitra]

GitHub user iraghumitra opened a pull request:

https://github.com/apache/metron/pull/768

Metron 1123: Add group by option using faceted search capabilities of 
metron-rest-api

## Contributor Comments
This PR extends the Metron Alerts GUI capabilities to do group by on Alerts 
Data

The PR is on top of METRON-1189

The group by buttons appear on top of the table. When user selects a group, 
the table view changes to a tree view with the values of the groups as the root 
of the tree. The individual alerts would be shown as the leaf nodes of the tree.

Tree view has all the following features

- Search will search through all the groups
- Users can click on the data  inside the table to search
- Users can select and table inside tree to view the details
- Users can escalate the alerts from the table inside the tree
- Tree view is refreshed automatically depending on the polling setting in 
the settings
- A cumulative score and number of alerts  in a group is shown next to 
group name
- Table inside a group can be paginated
- The column names for the alerts table in the tree can be configured via 
configure column pane

The table config settings is moved next to settings on the GUI as it makes 
more meaning there.


![image](https://user-images.githubusercontent.com/15019012/30694092-29135ce4-9ef0-11e7-88d4-06fee02040bb.png)

![image](https://user-images.githubusercontent.com/15019012/30694106-436d487a-9ef0-11e7-9f76-56b49ebd8c89.png)

![image](https://user-images.githubusercontent.com/15019012/30694113-4c7b8ad0-9ef0-11e7-9a58-5c7988f71430.png)

## Testing  

E2E test cases are available to test groups functionality. From root of the 
project 

```
cd metron-interface/metron-alerts
sh ./scripts/start-server-for-e2e.sh
```

In othere shell run the test cases

```
npm run e2e
```

All the features listed in contributor comments can be verified manually 


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iraghumitra/incubator-metron METRON-1123

Alternatively you can review and apply these