date:20171128

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/849
  
Instead of trying to explaining my position, I threw together a quick PR to 
illustrate my thoughts.  Feel free to take another approach.  I tested it with 
`g++` configured, unconfigured, and not in my PATH on macOS, as well as on a 
CentOS 7 with and without gcc installed, and an Ubuntu 14.04 box with an old 
version of gcc installed (not C++11 compliant).  All worked as expected.


---

[GitHub] metron issue #852: METRON-1239 Drop extra dev environments

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/852
  
+1 by inspection


---

[GitHub] metron issue #852: METRON-1239 Drop extra dev environments

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/852
  
Thanks @ottobackwards .  I just pushed that fix, plus some other little 
things I found.


---

Re: [DISCUSS] e2e test infrastructure

2017-11-28 Thread Otto Fowler

As long as there is not a large chuck of custom deployment that has to be
maintained docker sounds ideal.
I would like to understand what it would take to create the docker e2e env.



On November 28, 2017 at 17:27:13, Ryan Merriman (merrim...@gmail.com) wrote:

Currently the e2e tests for our Alerts UI depends on full dev being up and
running. This is not a good long term solution because it forces a
contributor/reviewer to run the tests manually with full dev running. It
would be better if the backend services could be made available to the e2e
tests while running in Travis. This would allow us to add the e2e tests to
our automated build process.

What is the right approach? Here are some options I can think of:

- Use the in-memory components we use for the backend integration tests
- Use a Docker approach
- Use mock components designed for the e2e tests

Mocking the backend would be my least favorite option because it would
introduce a complex module of code that we have to maintain.

The in-memory approach has some shortcomings but we may be able to solve
some of those by moving components to their own process and spinning them
up/down at the beginning/end of tests. Plus we are already using them.

My preference would be Docker because it most closely mimics a real
installation and gives you isolation, networking and dependency management
features OOTB. In many cases Dockerfiles are maintained and published by a
third party and require no work other than some setup like loading data or
templates/schemas. Elasticsearch is a good example.

I believe we could make any of these approaches work in Travis. What does
everyone think?

Ryan

[GitHub] metron issue #852: METRON-1239 Drop extra dev environments

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/852
  
I think you missed the inventory/quick-dev-platform stuff


---

[DISCUSS] e2e test infrastructure

2017-11-28 Thread Ryan Merriman

Currently the e2e tests for our Alerts UI depends on full dev being up and
running.  This is not a good long term solution because it forces a
contributor/reviewer to run the tests manually with full dev running.  It
would be better if the backend services could be made available to the e2e
tests while running in Travis.  This would allow us to add the e2e tests to
our automated build process.

What is the right approach?  Here are some options I can think of:

   - Use the in-memory components we use for the backend integration tests
   - Use a Docker approach
   - Use mock components designed for the e2e tests

Mocking the backend would be my least favorite option because it would
introduce a complex module of code that we have to maintain.

The in-memory approach has some shortcomings but we may be able to solve
some of those by moving components to their own process and spinning them
up/down at the beginning/end of tests.  Plus we are already using them.

My preference would be Docker because it most closely mimics a real
installation and gives you isolation, networking and dependency management
features OOTB.  In many cases Dockerfiles are maintained and published by a
third party and require no work other than some setup like loading data or
templates/schemas.  Elasticsearch is a good example.

I believe we could make any of these approaches work in Travis.  What does
everyone think?

Ryan

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

2017-11-28 Thread merrimanr

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/853

METRON-1337: List of facets should not be hardcoded

## Contributor Comments
This PR makes the list of facet fields in the Alerts UI configurable.
There is now a "search.facet.fields" setting in
https://github.com/apache/metron/blob/master/metron-interface/metron-rest/src/main/resources/application.yml
that is a comma-separate list of fields to be used as facets. Originally a
comment was made that the "host" field wasn't commonly used so I removed that
from the default list.

I can think of two options for exposing this configuration:
- through an AlertProfile that is specific to each user, stored in a
relational database and can be edited through REST
- a setting that is exposed in Ambari

I chose to include the first option in this PR to get the conversation
going. Is one of these preferable? The AlertProfile approach allows this
setting to be changed at runtime and each user has their own list of facet
fields. But it is not versioned like it would if it were in Ambari. Do we
prefer one over the other? Do we want both with Ambari being the default when
an AlertProfile doesn't exist for a user? Are there other options I'm not
thinking of?

This works similar to how default search indices work: it is managed in
the REST layer and can be overriden by including facet fields in a search
request. However it seemed useful to allow a way to explicitly NOT include
facets in a query so it works slightly different than indices. A missing
facetFields property in the request will use the defaults while an empty array
will disable facets. A missing indices property or an empty array will use the
default indices. Is this the correct behavior?

This has been tested in full dev and the UI e2e tests pass when run in
isolation. There is currently an effort to stabilize the e2e tests as a follow
on to https://github.com/apache/metron/pull/803 so I did not try to solve that
here.

I will add some documentation around configuring the facet field list and
facetFields behavior in a search request once we come to a consensus on the
solution.

Another issue I would like to point out. When I added facetFields to the
AlertProfile object it required a database update because that new field needed
to be added to the database. This would become an issue if someone were
upgrading from a previous version. Is this acceptable if we document it for
future upgrades? Is a relational database the right solution or should we
consider a more flexible storage option?

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).

- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?

### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```

- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via

[GitHub] metron pull request #852: METRON-1239 Drop extra dev environments

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/852

METRON-1239 Drop extra dev environments

Per the previous discussion referenced in JIRA, this PR drops support for 
the Quick Dev and Code Lab environments.  These have been effectively dead for 
quite a long time.

I tested this by performing a Full Dev deployment and Amazon EC2 deployment 
just to make sure I didn't delete anything that I should not have.

I'd really like to get the metron-deployments directory cleaned-up and 
reorganized to make it easier for new users to build and deploy Metron.  This 
is the first step.


### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1239

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/852.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #852


commit 4a7484abd26a941754f18d0b498aedb2c24d385f
Author: Nick Allen 
Date:   2017-11-28T19:44:58Z

METRON-1239 Drop extra dev environments




---

[GitHub] metron issue #847: METRON-1313: Update metron-deployment to use bro-pkg to i...

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/847
  
Looks good, Jon.  I agree with what you said.  Let's get the release 
process sorted before we commit this one.


---

[GitHub] metron issue #812: METRON-1273: Website documentation link should point to t...

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/812
  
Bump.  @JonZeolla You are good to go here.  Let me know if you need help 
merging/deploying the changes.


---

[GitHub] metron issue #830: METRON-1298: TimeRange Picker doesn't work on Safari

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/830
  
Hi @iraghumitra - Is this PR still valid?  Can you deconflict when you get 
some cycles?


---

[GitHub] metron pull request #846: METRON-1328: Enhance platform-info.sh script to ch...

2017-11-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/846


---

[GitHub] metron issue #836: METRON-1308: Fix Metron Documentation

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/836
  
+1  I like this @JonZeolla.  This reads more clearly IMHO.


---

[GitHub] metron issue #849: METRON-1334 Add C++11 Compliance Check to 'platform-info....

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/849
  
> Is this the expected output?

I agree with @ottobackwards . That looks like what I would expect to see.  

@JonZeolla I assume since you asked, it is not what you would expect.  What 
would prefer to see here?


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/851

METRON-1336 Patching Can Result in Bad Configuration

The following problems are addressed in this PR.

* A patch can be constructed that when applied creates an invalid 
configuration.  The invalid configuration is not discovered until attempting to 
"pull" it back out of Zookeeper.  In all cases, the result of applying a patch 
needs to be validated before pushing it to Zookeeper.

* The Profiler configuration can only be pushed when pushing all 
configurations at once.  Attempting to push just the Profiler configuration 
alone, fails silently.

* Added unit tests to validate that a 'bad' patch cannot be applied.

* Added a good number of unit tests to ensure invalid configurations cannot 
be pushed under all circumstances (parser, enrichment, indexing, profiler, 
"push all").

* Added a test to ensure that the Profiler configuration can be pushed 
independently.

## Manual Testing

1. Launch Full Dev.

1. Environment definitions that will be used later.
```
export METRON_HOME=/usr/metron/0.4.2
export ZOOKEEPER=node1:2181
```

1. Dump all existing configurations.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP
```

1. Dump just the Profiler configuration.  There should be none.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
```

1. Create a Profiler configuration on disk.
```
$ cat $METRON_HOME/config/profiler.json
{
  "profiles": [
{
  "profile": "hello-world",
  "onlyif":  "exists(ip_src_addr)",
  "foreach": "ip_src_addr",
  "init":{ "count": "0" },
  "update":  { "count": "count + 1" },
  "result":  "count"
}
  ]
}
```

1. Push the Profiler configuration.  
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PUSH -c PROFILER 
-i $METRON_HOME/config
```

1. Dump just the Profiler configuration.  It should be defined now.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
```

1. Create a patch for the Profiler configuration.
```
$ cat profile.patch
[
{
"op": "add",
"path": "/profiles/0/profile",
"value": "changed-profile-name"
}
]
```

1. Apply the patch.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c PROFILER 
-pf profile.patch
```

1. Dump the Profiler configuration and ensure that the name of the Profile 
was changed.
```
[root@node1 ~]# $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m 
DUMP -c PROFILER
  PROFILER Config: profiler
  {
"profiles" : [ {
  "profile" : "changed-profile-name",
  "onlyif" : "exists(ip_src_addr)",
  "foreach" : "ip_src_addr",
  "init" : {
"count" : "0"
  },
  "update" : {
"count" : "count + 1"
  },
  "result" : "count"
} ]
  }
```
  
1. Create a patch that would make the Profiler configuration invalid. 
```
[root@node1 ~]# cat bad.patch
  [
{
"op": "add",
"path": "/profiles/0/invalid",
"value": "22"
}
  ]
```

1.  Apply the bad patch.   The script should terminate with an error.
``` 
[root@node1 ~]# $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m 
PATCH -c PROFILER -pf bad.patch
2017-11-28 17:37:17 ERROR ConfigurationManager:314 - Unable to apply 
patch to Zookeeper config
java.lang.RuntimeException: Unable to load {
  "profiles" : [ {
"profile" : "changed-profile-name",
"onlyif" : "exists(ip_src_addr)",
"foreach" : "ip_src_addr",
"init" : {
  "count" : "0"
},
"update" : {
  "count" : "count + 1"
},
"result" : "count",
"invalid" : "22"
  } ]
}
at 
org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
at 
org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
at 
org.apache.metron.common.configuration.ConfigurationsUtils.applyConfigPatchToZookeeper(ConfigurationsUtils.java:621)
at

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

2017-11-28 Thread Nick Allen

I'll add a bit to Jon's technical comments.

* We only created a separate repo because it was a technical requirement to
leverage the bro-pkg mechanism.
* Leveraging the new bro-pkg mechanism has many advantages as outlined by
Jon.
* Enabling the bro-pkg mechanism is backwards compatible.  I can install
the plugin exactly how we use to.

While I agree with Jon's technical comments, I disagree with the
non-technical ones.

(1) I do not want to change our release management process.  While we
needed to make a new repo (a technical change), I did not want that to
change how we operate as a community (our procedures, policies, versioning
and release cycles).

(2) I see no value in adopting a separate release management process for
the Bro plugin alone.  Having a separate release process does not make the
plugin *more* available to the Bro community.

(3) I also see no value in positioning the plugin to be spun-out of the
Metron project.  It is a part of Metron and I want to see it benefit and
evolve "the Apache-way".

In my mind, the best way to accommodate the additional repo, while
minimizing changes to our release management process, is to treat the new
repo as a submodule.  I fail to see significant downsides to this
approach.  A few extract command-line options do not seem overly onerous to
me.

Many thanks go to Jon for all the hard work he has put in enhancing the
plugin.

On Mon, Nov 27, 2017 at 10:07 PM, zeo...@gmail.com  wrote:

> In an attempt to keep this from becoming unbearably long, I will try
> to keep my responses short, but I would be happy to elaborate.  That's a
> fairly good timeline and summary, but here are some clarifications in
> corresponding order:
>
> - The plugin history is quite short and you can probably get a good bit of
> context just by looking at the commits
> .
> - The plugin is only useful to the bro community, but it is rather popular.
> - The Bro team created the idea of bro packages, which can include bro
> plugins, bro scripts, or BroControl plugins.  So, instead of having a
> 'plugins' repo, they moved to have a 'packages' repo which is by default
> referenced by a bro-pkg tool they wrote for package management.
> - I believe I kicked this off (or at least I did in my head) when I
> started complaining
>  about the
> plugin divergence that occurred due to the move to bro/plugins (the right
> move at the time), but Metron's use of a local directory that hadn't been
> kept up to date.  My current efforts are an attempt to make sure this
> doesn't happen again, and to take advantage of the bro-pkg benefits.
> - The gap between ~3/31 and actual progress on 11/12 is completely on me -
> I had intentions of doing this work sooner but failed to do so.
> - You can most definitely still install/use the bro plugin without using
> bro-pkg.  In fact, the README in my PR
> 
> still has the instructions on how to do so.
>
> Q1:  The simple explanation is that the only thing that makes a plugin a
> bro package is the inclusion of a bro-pkg.meta file
> ,
> and it includes a build_command which could easily be manually performed to
> install by hand (assuming dependencies are met).
>
> I've worked with other projects that use submodules and while I'm fine
> discussing it, I suggest that we don't implement it.  I put together a
> quick example of why here[1], using the bro project as an example since
> it's top of mind.
>
> Q2:   I think the answer to Q1 answers this.  There is absolutely nothing
> stopping a git clone && cd $dir && configure && make && make install, but
> using bro-pkg to install/load takes into account dependencies
> 
>  and
> unit tests
> 
> when it is loaded (and thus fails early and more intuitively).  It only
> must be a separate repo (or, more technically correct, a git branch that
> includes only the package) because of how bro-pkg works.  If you'd like to
> get an idea of how this would work in application for Bro users, you can
> see my test instructions here
>  (specifically
> step #3).  If a 0.1 tag gets pushed to apache/metron-bro-plugin-kafka,
> the command could be `bro-pkg install metron-bro-plugin-kafka --version
> 0.1` or `bro-pkg install apache/metron-bro-plugin-kafka --version 0.1`
> due to this
>  (the
> --force is just to remove user interaction, for an ansible spin-up).
>
>
> 1:  To clone the Bro git repo, you must run `git clone

[GitHub] metron issue #847: METRON-1313: Update metron-deployment to use bro-pkg to i...

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/847
  
Thanks Otto.  Leaving this one open for now, until things get sorted on the 
mailing list.


---

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

2017-11-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/848


---

[GitHub] metron issue #848: METRON-1333 Ensure that ansible-docker can be used to bui...

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/848
  
Ahh sorry, missed that was in the docs as well.  Ok, looks good


---

[GitHub] metron issue #848: METRON-1333 Ensure that ansible-docker can be used to bui...

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/848
  
@JonZeolla I would like to change the readme example to include the 
quick_dev skip.  


---

[GitHub] metron issue #823: METRON-1286 Add MIN & MAX Stellar functions

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/823
  
I'm ok with the functions as they are, but if we are not going to verify 
the parameters, then remove the documented restriction.


---

[GitHub] metron issue #848: METRON-1333 Ensure that ansible-docker can be used to bui...