date:20170629

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/627

METRON-1007 ambari integration for metron-management

This pr integrates the metron-management jar file, and the stellar 
management functions it contains with the ambari installation, thus removing 
the requirement to hand deploy the jar file to the /usr/metron/VERSION/lib 
directory before using the functions.



###Testing
Spin up Full Dev

* verify that the jar has been deployed as such:
```bash
[vagrant@node1 ~]$ cd /usr/metron/0.4.0/
[vagrant@node1 0.4.0]$ ls
bin  config  flux  lib  patterns  web
[vagrant@node1 0.4.0]$ ls lib
metron-api-0.4.0.jar metron-data-management-0.4.0.jar 
metron-enrichment-0.4.0-uber.jar  metron-parsers-0.4.0-uber.jar  
metron-rest-0.4.0.jar
metron-common-0.4.0.jar  metron-elasticsearch-0.4.0-uber.jar  
metron-management-0.4.0.jar   metron-pcap-backend-0.4.0.jar
```

* run stellar and execute a stellar management function

```bash
[vagrant@node1 0.4.0]$ bin/stellar -z node1:2181
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
details.
Stellar, Go!
Please note that functions are loading lazily in the background and will be 
unavailable until loaded fully.
{es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, 
parser.error.topic=indexing}
[Stellar]>>> Functions loaded, you may refer to functions now...

[Stellar]>>> CONFIG_GET('PARSER','bro')
{
  "parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
  "sensorTopic":"bro",
  "parserConfig": {}
}
[Stellar]>>>
```

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [NA] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [NA] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [NA] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [NA] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron 
install_metron_managment_jar

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/627.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

GitHub user justinleet reopened a pull request:

https://github.com/apache/metron/pull/624

[DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maximum Time Limit

## Contributor Comments
Given that Apache travis fails more often, I'm using this PR to hopefully 
hit the skidoo clause more.  First step is adding more logging to see what's 
even happening when we kill slots.  Also calls a Thread.interrupt on the slot 
out of morbid curiosity.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/metron METRON-1004

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #624


commit bd484b5a0bede9214ec39a3c6f4192d9e948ed35
Author: justinjleet 
Date:   2017-06-23T13:15:02Z

first pass. Still need to fix storm kafka interaction + possibly cleanup

commit a470063422314711ec99a298d646202deda32fba
Author: justinjleet 
Date:   2017-06-23T16:04:53Z

fixes plus logging

commit 05e0a6468d283a2ee6a0ce4be8c551373be7f75b
Author: justinjleet 
Date:   2017-06-23T19:01:27Z

more stuffs

commit 05b9c1f0d04a76afb97c90c33d82268618129806
Author: justinjleet 
Date:   2017-06-24T17:49:52Z

more fixes and update

commit 912575eb92b2baf6f1334991372a883ad252d866
Author: justinjleet 
Date:   2017-06-24T18:37:15Z

removing error code

commit 11473f63841e18cef2fbb046c70c2e052b498b4f
Author: justinjleet 
Date:   2017-06-24T18:47:58Z

maybe making it work for superclasses like the parser tests.  Unsure if 
it'll maven

commit fa5c05205e4d5bf17d5ea665fd98711085391403
Author: justinjleet 
Date:   2017-06-25T00:18:00Z

Removing approximately 1M log errors by actually cleaning up ZK

commit a5b152e0e300640364fc28a1ae1bc80d7be4f628
Author: justinjleet 
Date:   2017-06-25T02:04:44Z

empty

commit

[GitHub] metron pull request #627: METRON-1007 ambari integration for metron-manageme...

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/627#discussion_r124873582
  
--- Diff: metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec 
---
@@ -409,6 +428,7 @@ chkconfig --del metron-management-ui
 # 
~~
 
 %changelog
+* Thu Jun 29 2017 Add Metron Management jar - 0.4.0+1
--- End diff --

Perhaps this should be:

```
* Thu Jun 29 2017 Otto Fowler  - 0.4.0+1
- Added Metron Management jar
```
Or similar?  Not clean on when we use individual names/contact info or 
`Apache Metron `.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/624#discussion_r124875780
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/components/ElasticSearchComponent.java
 ---
@@ -24,6 +24,8 @@
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthAction;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
+import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
--- End diff --

Because I tried something that didn't end up being useful, and didn't clean 
it up. I'll drop them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/620
  
Here's an idea to get us started.  A search request could look like:
```
{
  "query": "ip_src_addr:192.168.66.1",
  "from": 0,
  "size": 25,
  "sort": [
{ "timestamp": "desc"},
{ "some_other_field": "asc"}
  ]
}
```
where the query field is a lucene-style query (which is supported by both 
ES and Solr I believe).  A search response could look like:
```
{
  "total": 1,
  "results":
[
  {
"id": "1234abcd",
"score": 1.0,
"source": {
  "source:type": "bro",
  "host": "some host",
  "ip_src_addr": "192.168.66.1"
}
  },
  {
"id": "5678efgh",
"score": 1.0,
"source": {
  "source:type": "snort",
  "ip_src_port": 8080,
  "ip_src_addr": "192.168.66.1"
}
  }
  ]
}
```

We can add other fields (aggregations for example) as we add more features. 
 What do you think?  Is this enough to drive the current alerts UI or are there 
more fields needed?  Would it be helpful to add a 'filters' field to the 
request or can we assume the client-side code will handle adding filters to the 
lucene query?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...

GitHub user JonZeolla reopened a pull request:

https://github.com/apache/metron/pull/586

METRON-508 Expand Elasticsearch templates to support the standard bro logs

## Contributor Comments
This PR makes it easier for someone with an existing bro install to send 
some of their log files into Metron, based off of a combination of the [bro 
documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and 
a fresh install of bro 2.5.  There are future plans to expand on this via 
[METRON-518](https://issues.apache.org/jira/browse/METRON-518) and 
[METRON-908](https://issues.apache.org/jira/browse/METRON-908).  Specifically, 
this attempts to provide initial support the default-on fields of the following 
logs:
 - 
[Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info)
 - 
[DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info)
 - 
[FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info)
 - 
[Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info)
 - 
[CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo)
 - 
[SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info)
 - 
[SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info)
 - 
[Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info)
 - 
[Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info)
 - 
[DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info)
 - 
[SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info)
 - 
[Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info)
 - 
[Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info)
 - 
[X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info)
 - 
[DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo)


## Testing
1.  Create a working directory and pull in this PR
```
mkdir ~/metron-508
git clone https://github.com/apache/metron ~/metron-508/metron
cd ~/metron-508/metron
git remote add jonzeolla https://github.com/jonzeolla/metron
git pull jonzeolla METRON-508
```
1.  Modify 
[this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20)
 to remove `sensors,` (to spin up the real sensors).
```
sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
metron-deployment/vagrant/full-dev-platform/Vagrantfile
```
1.  Start up full-dev.
```
cd metron-deployment/vagrant/full-dev-platform
vagrant up
```
1.  Set up the environment in full-dev.
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin
service monit stop && service sensor-stubs stop bro && broctl stop
yum -y install jq wireshark
```
1.  Configure kafka in local.bro.
```
sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = 
set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, 
SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, 
Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' 
/usr/local/bro/share/bro/site/local.bro
echo "redef Kafka::debug = \"all\";" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Known::cert_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Software::asset_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
sed -i '86 a @load 
policy/protocols/dhcp/known-devices-and-hostnames.bro' 
/usr/local/bro/share/bro/site/local.bro
```
1.  Monitor the bro kafka topic
```
# Open a new terminal
cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
```
1.  Monitor the storm logs.
```
# Open a new terminal
cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
# Look at the storm logs (The "failed to parse" errors for ip_src_addr 
and ip_dst_addr are expected, and should be addressed as a part of METRON-939)
tail

[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...

Github user JonZeolla closed the pull request at:

https://github.com/apache/metron/pull/586


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #586: METRON-508 Expand Elasticsearch templates to support the ...

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/586
  
Last try before I merge in #624 and make it a dependancy.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...

GitHub user JonZeolla reopened a pull request:

https://github.com/apache/metron/pull/586

METRON-508 Expand Elasticsearch templates to support the standard bro logs

## Contributor Comments
This PR makes it easier for someone with an existing bro install to send 
some of their log files into Metron, based off of a combination of the [bro 
documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and 
a fresh install of bro 2.5.  There are future plans to expand on this via 
[METRON-518](https://issues.apache.org/jira/browse/METRON-518) and 
[METRON-908](https://issues.apache.org/jira/browse/METRON-908).  Specifically, 
this attempts to provide initial support the default-on fields of the following 
logs:
 - 
[Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info)
 - 
[DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info)
 - 
[FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info)
 - 
[Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info)
 - 
[CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo)
 - 
[SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info)
 - 
[SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info)
 - 
[Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info)
 - 
[Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info)
 - 
[DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info)
 - 
[SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info)
 - 
[Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info)
 - 
[Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info)
 - 
[X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info)
 - 
[DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo)


## Testing
1.  Create a working directory and pull in this PR
```
mkdir ~/metron-508
git clone https://github.com/apache/metron ~/metron-508/metron
cd ~/metron-508/metron
git remote add jonzeolla https://github.com/jonzeolla/metron
git pull jonzeolla METRON-508
```
1.  Modify 
[this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20)
 to remove `sensors,` (to spin up the real sensors).
```
sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
metron-deployment/vagrant/full-dev-platform/Vagrantfile
```
1.  Start up full-dev.
```
cd metron-deployment/vagrant/full-dev-platform
vagrant up
```
1.  Set up the environment in full-dev.
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin
service monit stop && service sensor-stubs stop bro && broctl stop
yum -y install jq wireshark
```
1.  Configure kafka in local.bro.
```
sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = 
set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, 
SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, 
Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' 
/usr/local/bro/share/bro/site/local.bro
echo "redef Kafka::debug = \"all\";" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Known::cert_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Software::asset_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
sed -i '86 a @load 
policy/protocols/dhcp/known-devices-and-hostnames.bro' 
/usr/local/bro/share/bro/site/local.bro
```
1.  Monitor the bro kafka topic
```
# Open a new terminal
cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
```
1.  Monitor the storm logs.
```
# Open a new terminal
cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
# Look at the storm logs (The "failed to parse" errors for ip_src_addr 
and ip_dst_addr are expected, and should be addressed as a part of METRON-939)
tail

[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...

Github user JonZeolla closed the pull request at:

https://github.com/apache/metron/pull/586


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/624#discussion_r124869167
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/components/ElasticSearchComponent.java
 ---
@@ -24,6 +24,8 @@
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthAction;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
+import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
--- End diff --

why were the imports added?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user justinleet closed the pull request at:

https://github.com/apache/metron/pull/624


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

GitHub user justinleet reopened a pull request:

https://github.com/apache/metron/pull/624

[DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maximum Time Limit

## Contributor Comments
This PR cleans up and speeds up various issues we've seen on Travis.

- This should avoid the skadoo, by just outright stopping problematic 
threads.
- Allows for the reuse of some of the InMemoryComponents, to avoid a lot of 
the spinup time in tests
- Improvements to a variety of tests to make them faster
- Refactoring of some parser integration tests to be less integrationy and 
more unit test-ish.  Specifically because not every parser needs a full 
integration test and they take awhile.
- Moving to VM in Travis
- More details as provided in comments

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/metron METRON-1004

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #624


commit bd484b5a0bede9214ec39a3c6f4192d9e948ed35
Author: justinjleet 
Date:   2017-06-23T13:15:02Z

first pass. Still need to fix storm kafka interaction + possibly cleanup

commit a470063422314711ec99a298d646202deda32fba
Author: justinjleet 
Date:   2017-06-23T16:04:53Z

fixes plus logging

commit 05e0a6468d283a2ee6a0ce4be8c551373be7f75b
Author: justinjleet 
Date:   2017-06-23T19:01:27Z

more stuffs

commit 05b9c1f0d04a76afb97c90c33d82268618129806
Author: justinjleet 
Date:   2017-06-24T17:49:52Z

more fixes and update

commit 912575eb92b2baf6f1334991372a883ad252d866
Author: justinjleet 
Date:   2017-06-24T18:37:15Z

removing error code

commit 11473f63841e18cef2fbb046c70c2e052b498b4f
Author: justinjleet 
Date:   2017-06-24T18:47:58Z

maybe making it work for superclasses like the parser tests.  Unsure if 
it'll maven

commit

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/530
  
Ok, I was getting ready to bail on this for the day but couldn't stop. I 
tried one more thing - setting the maven-archetype-plugin version to **2.4** 
and it works as expected. Something must be broken in 3.0.0, but I didn't see 
any jiras after a quick search. I also tried 3.0.1 and had the same problem. 
Would you be opposed to changing the plugin version to 2.4? Note that v2.7 also 
had some other problems with files not being included/excluded properly. 2.4 
seems to work well afaict.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] Apache Metron 0.4.0 release

2017-06-29 Thread James Sirota

+1 (Binding)

* Verified Keys
* Verified mvn clean install completed successfully
* Verified AWS install of core via Mpack


29.06.2017, 09:14, "Justin Leet" :
> +1 (Non-binding)
>
> * Verified Keys
> * Verified mvn clean install completed successfully
> * Ran full dev: saw data flow through, ran a couple of the REST APIs, and
> opened up and clicked through a bit of the Management API.
> * Examined site-book and didn't see any issues
>
> On Thu, Jun 29, 2017 at 11:46 AM, Casey Stella  wrote:
>
>>  +1 (binding)
>>  * Verified keys
>>  * Verified mvn build
>>  * Verified unit and integration tests run
>>  * Verified license check runs
>>  * Verified fulldev spun up with smoketest
>>
>>  On Wed, Jun 28, 2017 at 8:10 PM, Anand Subramanian <
>>  asubraman...@hortonworks.com> wrote:
>>
>>  > +1 (non-binding)
>>  >
>>  > * Brought up Metron stack on 12-node CentOS7 openstack cluster
>>  > * Verify all services come up fine [PASS]
>>  > * Bro, YAF and snort - ingest into respective kafka topics and write
>>  > indices [PASS]
>>  > * Add squid telemetry, ingest into kafka topic and write indices [PASS]
>>  > * Metron YAF Zeppelin dashboard with sample ingested YAF data [PASS]
>>  > * Management UI and REST Swagger UI sanity check [PASS]
>>  >
>>  >
>>  > -Anand
>>  >
>>  >
>>  >
>>  >
>>  >
>>  > On 6/28/17, 12:06 AM, "Matt Foley"  wrote:
>>  >
>>  > >This is a call to vote on releasing this rc4 as “Apache Metron 0.4.0”.
>>  > >(Note: this is rc4 because the release candidate needed to be modified
>>  > with another commit after the rc3 tag was pushed to public.)
>>  > >
>>  > >Full list of changes in this release:
>>  > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/RELEASE_NOTES
>>  > >
>>  > >The tag/commit to be voted upon is:
>>  > >d52f574f8294e453ecad3871526858a0c3c2033d (tag apache-metron-0.4.0-rc4)
>>  > >
>>  > >The source archive being voted upon can be found here:
>>  > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-
>>  > RC4/apache-metron-0.4.0-rc4.tar.gz
>>  > >and in github at:
>>  > >https://github.com/apache/metron/tree/Metron_0.4.0
>>  > >
>>  > >Other release files, signatures and digests can be found here:
>>  > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS
>>  > >
>>  > >The release artifacts are signed with the following key:
>>  > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS
>>  > >pub rsa4096/4169AA27ECB31663 2011-07-31 [SCEA]
>>  > >Key fingerprint = 7854 36A7 8258 6B71 829C 67A0 4169 AA27 ECB3 1663
>>  > >uid = Matthew Foley (CODE SIGNING KEY) 
>>  > >
>>  > >Please vote on releasing this package as Apache Metron 0.4.0.
>>  > >When voting, please list the actions taken to verify the release.
>>  > >
>>  > >Recommended build validation and verification instructions are posted
>>  > here:
>>  > >https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>>  > >
>>  > >This vote will be open for at least 72 hours. Please vote one of the
>>  > following responses:
>>  > >+1 Release this package as Apache Metron 0.4.0-RC4
>>  > >0 No opinion
>>  > >-1 Do not release this package because...
>>  > >
>>  > >Thank you,
>>  > >--Matt
>>  > >(your friendly release manager)
>>  > >
>>  > >
>>  > >
>>  >

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/624#discussion_r124927666
  
--- Diff: 
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/KafkaControllerIntegrationTest.java
 ---
@@ -61,6 +62,7 @@
   private static final int KAFKA_RETRY = 10;
   @Autowired
   private KafkaComponent kafkaWithZKComponent;
+  private ComponentRunner runner;
--- End diff --

I don't think this is that critical to getting us back to a working travis 
build.  Just my opinion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
I'm ok with that, I would feel better if we got a tie breaking opinion 
though.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

GitHub user justinleet reopened a pull request:

https://github.com/apache/metron/pull/624

[DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maximum Time Limit

## Contributor Comments
This PR cleans up and speeds up various issues we've seen on Travis.

- This should avoid the skadoo, by just outright stopping problematic 
threads.
- Allows for the reuse of some of the InMemoryComponents, to avoid a lot of 
the spinup time in tests
- Improvements to a variety of tests to make them faster
- Refactoring of some parser integration tests to be less integrationy and 
more unit test-ish.  Specifically because not every parser needs a full 
integration test and they take awhile.
- Moving to VM in Travis
- More details as provided in comments

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/metron METRON-1004

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #624


commit bd484b5a0bede9214ec39a3c6f4192d9e948ed35
Author: justinjleet 
Date:   2017-06-23T13:15:02Z

first pass. Still need to fix storm kafka interaction + possibly cleanup

commit a470063422314711ec99a298d646202deda32fba
Author: justinjleet 
Date:   2017-06-23T16:04:53Z

fixes plus logging

commit 05e0a6468d283a2ee6a0ce4be8c551373be7f75b
Author: justinjleet 
Date:   2017-06-23T19:01:27Z

more stuffs

commit 05b9c1f0d04a76afb97c90c33d82268618129806
Author: justinjleet 
Date:   2017-06-24T17:49:52Z

more fixes and update

commit 912575eb92b2baf6f1334991372a883ad252d866
Author: justinjleet 
Date:   2017-06-24T18:37:15Z

removing error code

commit 11473f63841e18cef2fbb046c70c2e052b498b4f
Author: justinjleet 
Date:   2017-06-24T18:47:58Z

maybe making it work for superclasses like the parser tests.  Unsure if 
it'll maven

commit

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user justinleet closed the pull request at:

https://github.com/apache/metron/pull/624


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user justinleet closed the pull request at:

https://github.com/apache/metron/pull/624


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/624#discussion_r124921605
  
--- Diff: 
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/KafkaControllerIntegrationTest.java
 ---
@@ -148,6 +150,15 @@ public void stop() {
 
   @Before
   public void setup() throws Exception {
+runner = new ComponentRunner.Builder()
+.withComponent("kafka", kafkaWithZKComponent)
+.withCustomShutdownOrder(new String[]{"kafka"})
+.build();
+try {
+  runner.start();
+} catch (UnableToStartException e) {
+  e.printStackTrace();
--- End diff --

Also, we should throw a runtime exception here, probably.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
@mmiklavc can you comment with what your pom looks like now?
Why does 3.0 work on my machine though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/530

@ottobackwards removing the extra project element allows me to build the
project now. However, I'm still unable to use the archetype. It neither shows
up in the local catalog listing nor allows me to enter the info directly. It
fails in both cases, even though the archetype is now appearing in
`~/.m2/repository/archetype-catalog.xml`. Per the Maven docs for the archetype
plugin, the entries should be in `~/.m2/archetype-catalog.xml`. -
https://maven.apache.org/archetype/maven-archetype-plugin/specification/archetype-catalog.html.

The only thing I could do to get this working was to explicitly update the
local catalog with the archetype plugin, which doesn't make any sense to me
seeing as this is by default tied to the install goal in the first place.

There are 2 archetype-catalog.xml files locally with Maven and I'm unclear
as to why. The only difference appears to be that 1) only
~/.m2/archetype-catalog.xml entries show up when you run `mvn
archetype:generate -DarchetypeCatalog=local` and 2) non-SNAPSHOT entries end up
in `~/.m2/archetype-catalog.xml` whereas SNAPSHOT entries end up in
`~/.m2/repository/archetype-catalog.xml`.

```
cat ~/.m2/repository/archetype-catalog.xml

http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0
http://maven.apache.org/xsd/archetype-catalog-1.0.0.xsd;

xmlns="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>

org.apache.metron
metron-maven-parser-extension-archetype
0.4.0
Apache Maven Parser Extension Archetype for
Metron

```
When I run `mvn archetype:update-local-catalog` the entry appears in
~/.m2/archetype-catalog.xml. If I delete the entry and re-run that command, the
entry reappears. Now when I run the archetype:generate command, the Metron
archetype appears as expected. Is anyone else seeing this behavior? This does
not happen for me when I install and attempt to use other archetypes, e.g.
https://github.com/mmiklavc/hadoop-bootstrap-archetype

Here's the output, as desired, after the archetype is added to the correct
archetype-catalog.xml

```
[INFO] No archetype defined. Using maven-archetype-quickstart
(org.apache.maven.archetypes:maven-archetype-quickstart:1.0)
Choose archetype:
...
4: local -> com.michaelmiklavcic:hadoop-bootstrap-archetype (Archetype -
hadoop-bootstrap-archetype)
5: local -> org.apache.metron:metron-maven-archetypes (Apache Maven
Archetypes for Metron)
```

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/624#discussion_r124922704
  
--- Diff: 
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/KafkaControllerIntegrationTest.java
 ---
@@ -61,6 +62,7 @@
   private static final int KAFKA_RETRY = 10;
   @Autowired
   private KafkaComponent kafkaWithZKComponent;
+  private ComponentRunner runner;
--- End diff --

I don't think so.  `setup()` uses kafkaWithZKComponent, which is 
`@Autowired`.  I don't think that can be static, so there's probably more 
refactoring involved.  I can look at it, if it you want.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maxi...

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/624
  
Most recent commits attempt to fix what appears to be a preexisting 
intermittent test issue regarding Kafka.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/624#discussion_r124921122
  
--- Diff: 
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/KafkaControllerIntegrationTest.java
 ---
@@ -61,6 +62,7 @@
   private static final int KAFKA_RETRY = 10;
   @Autowired
   private KafkaComponent kafkaWithZKComponent;
+  private ComponentRunner runner;
--- End diff --

Can we make this and `setup()` static and make the setup/teardown be 
@BeforeClass/@AfterClass?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/624#discussion_r124922443
  
--- Diff: 
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/KafkaControllerIntegrationTest.java
 ---
@@ -148,6 +150,15 @@ public void stop() {
 
   @Before
   public void setup() throws Exception {
+runner = new ComponentRunner.Builder()
+.withComponent("kafka", kafkaWithZKComponent)
+.withCustomShutdownOrder(new String[]{"kafka"})
+.build();
+try {
+  runner.start();
+} catch (UnableToStartException e) {
+  e.printStackTrace();
--- End diff --

I'd be ok with that.  This will just mask an error until it fails later 
less explicably.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/530
  
Archetype plugin version is the only change

```


http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
  4.0.0

  org.apache.metron
  metron-maven-parser-extension-archetype
  0.4.0
  maven-archetype

  metron-maven-parser-extension-archetype

  

  
org.apache.maven.archetype
archetype-packaging
3.0.0
  



  

  maven-archetype-plugin
  2.4

  

  

  Apache Maven Parser Extension Archetype for 
Metron

```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...

GitHub user JonZeolla reopened a pull request:

https://github.com/apache/metron/pull/586

METRON-508 Expand Elasticsearch templates to support the standard bro logs

## Contributor Comments
This PR makes it easier for someone with an existing bro install to send 
some of their log files into Metron, based off of a combination of the [bro 
documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and 
a fresh install of bro 2.5.  There are future plans to expand on this via 
[METRON-518](https://issues.apache.org/jira/browse/METRON-518) and 
[METRON-908](https://issues.apache.org/jira/browse/METRON-908).  Specifically, 
this attempts to provide initial support the default-on fields of the following 
logs:
 - 
[Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info)
 - 
[DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info)
 - 
[FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info)
 - 
[Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info)
 - 
[CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo)
 - 
[SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info)
 - 
[SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info)
 - 
[Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info)
 - 
[Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info)
 - 
[DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info)
 - 
[SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info)
 - 
[Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info)
 - 
[Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info)
 - 
[X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info)
 - 
[DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo)


## Testing
1.  Create a working directory and pull in this PR
```
mkdir ~/metron-508
git clone https://github.com/apache/metron ~/metron-508/metron
cd ~/metron-508/metron
git remote add jonzeolla https://github.com/jonzeolla/metron
git pull jonzeolla METRON-508
```
1.  Modify 
[this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20)
 to remove `sensors,` (to spin up the real sensors).
```
sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
metron-deployment/vagrant/full-dev-platform/Vagrantfile
```
1.  Start up full-dev.
```
cd metron-deployment/vagrant/full-dev-platform
vagrant up
```
1.  Set up the environment in full-dev.
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin
service monit stop && service sensor-stubs stop bro && broctl stop
yum -y install jq wireshark
```
1.  Configure kafka in local.bro.
```
sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = 
set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, 
SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, 
Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' 
/usr/local/bro/share/bro/site/local.bro
echo "redef Kafka::debug = \"all\";" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Known::cert_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Software::asset_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
sed -i '86 a @load 
policy/protocols/dhcp/known-devices-and-hostnames.bro' 
/usr/local/bro/share/bro/site/local.bro
```
1.  Monitor the bro kafka topic
```
# Open a new terminal
cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
```
1.  Monitor the storm logs.
```
# Open a new terminal
cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
# Look at the storm logs (The "failed to parse" errors for ip_src_addr 
and ip_dst_addr are expected, and should be addressed as a part of METRON-939)
tail

[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...

Github user JonZeolla closed the pull request at:

https://github.com/apache/metron/pull/586


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #626: METRON-947 [DO NOT MERGE UNTIL METRON-942] Stellar...

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/626

METRON-947 [DO NOT MERGE UNTIL METRON-942] Stellar Management Function for 
Metron Parser Extension Configurations

METRON-777 adds the Metron Extension capability, the default parsers as 
extensions, and the parser archetype
METRON-942 adds metron-rest support for installing and uninstalling parser 
extensions, as well as the configuration primitives for extensions in zookeeper

This PR adds a Stellar Management command to retrieve those configurations.

## Testing

### Build and install some parser extensions 

Follow the test plan for 
[METRON-942](https://github.com/apache/metron/pull/580), to the point where you 
have built and installed one or more parser extensions.

### deploy the metron-management jar per readme

### Run Stellar
```bash
[vagrant@node1 ~]$ /usr/metron/0.4.0/bin/stellar  -z node1:2181
```
Run the command:

```bash
CONFIG_GET('PARSER_EXTENSION', 'metron-parser-nice-assembly-0_4_0')
```
verify that the configuration return matches what you would see in the rest 
client
if you executed  Parser Extension Controller : get 
/api/v1/ext/parsers/{name} -> name = metron-parser-nice-assembly-0_4_0


### For all changes:
- [ x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [na ] Have you included steps to reproduce the behavior or problem that 
is being changed or addressed?
- [ x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ na] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron METRON-947

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/626.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #626


commit 864d320d91c522dfc2eb63fc12341f316a3f8952
Author: Otto Fowler 
Date:   2017-03-17T04:56:49Z

Metron Extension system

Based on Apache Nifi Nars

NAR changes
* new lib , rebrand to bundles from NAR
* port to VFS/FileObject from File based
* ability to set property values
* Rework FileUtils so that you can derive and override
* added initializers to set 'classes' that we care about instead of hard 
coding them, still needs defaults
* added components nec. for integration tests ( do not want dep. on 
metron-* )
* VFSClassloader for NarClassLoader
* Hdfs based integration test version of unpacknars tests
* HDFS ( filesystem ) based fileutilities to cover for writes to hdfs, 
since VFS is currently R/O HDFS
* modified plugin to support configuration of outputs
* use class index not service loader ( both subclass and annotated 
supported )

Archetype
* Parser Extension archetyp
* incudes all configuration
* creates tar.gz with bundle and configuration
* class index

Re: [VOTE] Apache Metron 0.4.0 release

2017-06-29 Thread Justin Leet

+1 (Non-binding)

* Verified Keys
* Verified mvn clean install completed successfully
* Ran full dev: saw data flow through, ran a couple of the REST APIs, and
opened up and clicked through a bit of the Management API.
* Examined site-book and didn't see any issues

On Thu, Jun 29, 2017 at 11:46 AM, Casey Stella  wrote:

> +1 (binding)
> * Verified keys
> * Verified mvn build
> * Verified unit and integration tests run
> * Verified license check runs
> * Verified fulldev spun up with smoketest
>
> On Wed, Jun 28, 2017 at 8:10 PM, Anand Subramanian <
> asubraman...@hortonworks.com> wrote:
>
> > +1 (non-binding)
> >
> > * Brought up Metron stack on 12-node CentOS7 openstack cluster
> > * Verify all services come up fine [PASS]
> > * Bro, YAF and snort - ingest into respective kafka topics and write
> > indices [PASS]
> > * Add squid telemetry, ingest into kafka topic and write indices [PASS]
> > * Metron YAF Zeppelin dashboard with sample ingested YAF data [PASS]
> > * Management UI and REST Swagger UI sanity check [PASS]
> >
> >
> > -Anand
> >
> >
> >
> >
> >
> > On 6/28/17, 12:06 AM, "Matt Foley"  wrote:
> >
> > >This is a call to vote on releasing this rc4 as “Apache Metron 0.4.0”.
> > >(Note: this is rc4 because the release candidate needed to be modified
> > with another commit after the rc3 tag was pushed to public.)
> > >
> > >Full list of changes in this release:
> > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/RELEASE_NOTES
> > >
> > >The tag/commit to be voted upon is:
> > >d52f574f8294e453ecad3871526858a0c3c2033d (tag apache-metron-0.4.0-rc4)
> > >
> > >The source archive being voted upon can be found here:
> > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-
> > RC4/apache-metron-0.4.0-rc4.tar.gz
> > >and in github at:
> > >https://github.com/apache/metron/tree/Metron_0.4.0
> > >
> > >Other release files, signatures and digests can be found here:
> > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS
> > >
> > >The release artifacts are signed with the following key:
> > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS
> > >pub   rsa4096/4169AA27ECB31663 2011-07-31 [SCEA]
> > >Key fingerprint = 7854 36A7 8258 6B71 829C  67A0 4169 AA27 ECB3 1663
> > >uid = Matthew Foley (CODE SIGNING KEY) 
> > >
> > >Please vote on releasing this package as Apache Metron 0.4.0.
> > >When voting, please list the actions taken to verify the release.
> > >
> > >Recommended build validation and verification instructions are posted
> > here:
> > >https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> > >
> > >This vote will be open for at least 72 hours.  Please vote one of the
> > following responses:
> > >+1 Release this package as Apache Metron 0.4.0-RC4
> > >0 No opinion
> > >-1 Do not release this package because...
> > >
> > >Thank you,
> > >--Matt
> > >(your friendly release manager)
> > >
> > >
> > >
> >
>

[GitHub] metron issue #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maxi...

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/624
  
Oh, we should also spin this up on full-dev and ensure that any of the 
changes to the pom files didn't screw anything up.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maxi...

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/624
  
The latest commit moves to a VM instead of using a container during the 
Travis build.  Testing on local Travis, this seemed to produce more consistent 
results that didn't have the same wide variance.  Several runs clustered inside 
the 25m-30m mark, and I haven't seen one outside that range.

It still needs to be run several times to make sure there won't be issues 
with the main Apache infra.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/624#discussion_r124842566
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserDriver.java
 ---
@@ -0,0 +1,168 @@
+/**
--- End diff --

readme about parser committed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user justinleet closed the pull request at:

https://github.com/apache/metron/pull/624


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/624#discussion_r124821164
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserDriver.java
 ---
@@ -0,0 +1,168 @@
+/**
--- End diff --

As I see it, there are a couple of things we want to test for every parser:
* The parser works (i.e. the unit tests we currently have)
* The parser can be driven from within the `ParserBolt` (the current 
integration tests as of this PR)

Beyond that, I think it's reasonable to have a `ParserIntegrationTest` 
which just validates data flows through the parser topology assuming a 
correctly working parser and just validates that things work within Storm.  I 
would not expect every parser to require this kind of test.  We have something 
similar to that in `SimpleHbaseEnrichmentWriterIntegrationTest`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] Apache Metron 0.4.0 release

2017-06-29 Thread Casey Stella

+1 (binding)
* Verified keys
* Verified mvn build
* Verified unit and integration tests run
* Verified license check runs
* Verified fulldev spun up with smoketest

On Wed, Jun 28, 2017 at 8:10 PM, Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> +1 (non-binding)
>
> * Brought up Metron stack on 12-node CentOS7 openstack cluster
> * Verify all services come up fine [PASS]
> * Bro, YAF and snort - ingest into respective kafka topics and write
> indices [PASS]
> * Add squid telemetry, ingest into kafka topic and write indices [PASS]
> * Metron YAF Zeppelin dashboard with sample ingested YAF data [PASS]
> * Management UI and REST Swagger UI sanity check [PASS]
>
>
> -Anand
>
>
>
>
>
> On 6/28/17, 12:06 AM, "Matt Foley"  wrote:
>
> >This is a call to vote on releasing this rc4 as “Apache Metron 0.4.0”.
> >(Note: this is rc4 because the release candidate needed to be modified
> with another commit after the rc3 tag was pushed to public.)
> >
> >Full list of changes in this release:
> >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/RELEASE_NOTES
> >
> >The tag/commit to be voted upon is:
> >d52f574f8294e453ecad3871526858a0c3c2033d (tag apache-metron-0.4.0-rc4)
> >
> >The source archive being voted upon can be found here:
> >https://dist.apache.org/repos/dist/dev/metron/0.4.0-
> RC4/apache-metron-0.4.0-rc4.tar.gz
> >and in github at:
> >https://github.com/apache/metron/tree/Metron_0.4.0
> >
> >Other release files, signatures and digests can be found here:
> >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS
> >
> >The release artifacts are signed with the following key:
> >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS
> >pub   rsa4096/4169AA27ECB31663 2011-07-31 [SCEA]
> >Key fingerprint = 7854 36A7 8258 6B71 829C  67A0 4169 AA27 ECB3 1663
> >uid = Matthew Foley (CODE SIGNING KEY) 
> >
> >Please vote on releasing this package as Apache Metron 0.4.0.
> >When voting, please list the actions taken to verify the release.
> >
> >Recommended build validation and verification instructions are posted
> here:
> >https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> >
> >This vote will be open for at least 72 hours.  Please vote one of the
> following responses:
> >+1 Release this package as Apache Metron 0.4.0-RC4
> >0 No opinion
> >-1 Do not release this package because...
> >
> >Thank you,
> >--Matt
> >(your friendly release manager)
> >
> >
> >
>

[GitHub] metron issue #586: METRON-508 Expand Elasticsearch templates to support the ...

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/586
  
@nickwallen @justinleet Can I get a quick review of my recent changes 
before I merge?  Mostly focused on how I approached documenting the template, 
but feel free to comment on anything.

I just spun this up in full-dev and verified that my above steps are still 
valid and this works as intended.  At this point I'm iteratively crossing my 
fingers and retrying Travis until it gets the whole way through and gives me a 
result.  Can't wait for #624


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maxi...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/624
  
I'm spinning it up now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/624#discussion_r124821374
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserDriver.java
 ---
@@ -0,0 +1,168 @@
+/**
--- End diff --

I can provide a readme about parser testing, yep.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maxi...

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/624
  
The commits `6824ca9` through `1c98ae4` above represent the following:
* `ProfilerHBaseClientTest` (using mock HTable rather than hbase)
* `HBaseClientTest` (deletes contents, rather than drops table between 
testcases)
* The `StixExtractorTest` (multithreading the cases)
* Config Test in data management (reusing zookeeper instance across class)
* `FSFunctionsTest` (reusing filesystem objects)

The remainder of the commits are around setting the NPM cache to be cached 
in travis.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r124780191
  
--- Diff: 
metron-maven-archetypes/metron-maven-parser-extension-archetype/src/main/resources/archetype-resources/metron-parser-__parserName__-bundle/pom.xml
 ---
@@ -0,0 +1,53 @@
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+${groupId}
+${rootArtifactId}
+${version}
+
+
+metron-parser-${parserName}-bundle
+${version}
+metron-parser-${parserName}-bundle
+bundle
+
+true
+false
+
+
+
+
+${groupId}
+metron-parser-${parserName}
+${version}
+
+
+
+
+
+
+org.apache.metron
+bundles-maven-plugin
+${metronVersion}
+true
+
+
+
+
--- End diff --

awesome find!  thanks - all taken care of.  Let me know if it resolves 3.5?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r124779193
  
--- Diff: metron-maven-archetypes/README.md ---
@@ -0,0 +1,56 @@
+# Metron Maven Archetypes
+
+These are [Apache Maven 
Archetypes](http://maven.apache.org/archetype/index.html) for use in creating 
extension components for the Apache Metron system
+Archetypes are helpful in creating maven based projects with the correct 
setup and dependency configurations for the target system, including providing 
sample implementations.
+
+### metron-maven-parser-extension-archetype
+This is an archetype for creating an Apache Metron Parser Extension
+
+ Use
+
+Build and install the archetype
+```
+$ mvn install
+```
+
+Create a directory to host your extension code
+
+```
+$ mkdir ~/src/my-parser-extension
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maxi...

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/624
  
travis passed. See https://travis-ci.org/apache/metron/builds/248356870

I'm going to kick it up and down a few times.  Hopefully it'll start 
showing here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

GitHub user justinleet reopened a pull request:

https://github.com/apache/metron/pull/624

[DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maximum Time Limit

## Contributor Comments
Given that Apache travis fails more often, I'm using this PR to hopefully 
hit the skidoo clause more.  First step is adding more logging to see what's 
even happening when we kill slots.  Also calls a Thread.interrupt on the slot 
out of morbid curiosity.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/metron METRON-1004

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #624


commit bd484b5a0bede9214ec39a3c6f4192d9e948ed35
Author: justinjleet 
Date:   2017-06-23T13:15:02Z

first pass. Still need to fix storm kafka interaction + possibly cleanup

commit a470063422314711ec99a298d646202deda32fba
Author: justinjleet 
Date:   2017-06-23T16:04:53Z

fixes plus logging

commit 05e0a6468d283a2ee6a0ce4be8c551373be7f75b
Author: justinjleet 
Date:   2017-06-23T19:01:27Z

more stuffs

commit 05b9c1f0d04a76afb97c90c33d82268618129806
Author: justinjleet 
Date:   2017-06-24T17:49:52Z

more fixes and update

commit 912575eb92b2baf6f1334991372a883ad252d866
Author: justinjleet 
Date:   2017-06-24T18:37:15Z

removing error code

commit 11473f63841e18cef2fbb046c70c2e052b498b4f
Author: justinjleet 
Date:   2017-06-24T18:47:58Z

maybe making it work for superclasses like the parser tests.  Unsure if 
it'll maven

commit fa5c05205e4d5bf17d5ea665fd98711085391403
Author: justinjleet 
Date:   2017-06-25T00:18:00Z

Removing approximately 1M log errors by actually cleaning up ZK

commit a5b152e0e300640364fc28a1ae1bc80d7be4f628
Author: justinjleet 
Date:   2017-06-25T02:04:44Z

empty

commit

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r124722777
  
--- Diff: 
metron-maven-archetypes/metron-maven-parser-extension-archetype/src/main/resources/archetype-resources/metron-parser-__parserName__-bundle/pom.xml
 ---
@@ -0,0 +1,53 @@
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+${groupId}
+${rootArtifactId}
+${version}
+
+
+metron-parser-${parserName}-bundle
+${version}
+metron-parser-${parserName}-bundle
+bundle
+
+true
+false
+
+
+
+
+${groupId}
+metron-parser-${parserName}
+${version}
+
+
+
+
+
+
+org.apache.metron
+bundles-maven-plugin
+${metronVersion}
+true
+
+
+
+
--- End diff --

The extra project element is what is causing problems in Maven 3.5 and not 
3.3.9. They must have changed how strict the parsing is.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maxi...

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/624
  
The final commits are intended to migrate the parser integration tests into 
a driver that just executes the bolt without storm or kafka.  This should save 
minutes of time and set us up for new parsers to not require such intense 
testing harnesses that blow up the build times.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...

Github user JonZeolla closed the pull request at:

https://github.com/apache/metron/pull/586


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #580: METRON-942 [NO MERGE UNTIL METRON-777] Rest api an...

GitHub user ottobackwards reopened a pull request:

https://github.com/apache/metron/pull/580

METRON-942 [NO MERGE UNTIL METRON-777] Rest api and configuration for 
Metron parser extensions

## Contributor Comments
METRON-777 introduces metron extensions and metron parser extensions. The 
existing parsers have been moved to be packaged and loaded as extensions.
A maven archetype for creating parser extensions has also been created.

What is missing however is the ability to install a parser extension 
created outside the metron tree by the archetype into a metron system.
This PR introduces a REST api to accomplish this.

Along with that, this PR introduces a parser extension configuration type, 
for zookeeper that serves as a registry for installed extensions
and their default configurations.  The idea being that in a follow on 
effort, the management UI will create a new 'instance' of a parser type
through cloning, editing and deploying these configurations.  The included 
configurations are indexing, enrichment, parser, and elasticsearch.
There may be multiple of these configurations, as each parser extension 
library can contain more than one parser.

NOTE: the elasticsearch template is NOT deployed to elasticsearch at this 
time. Any parser created will still need a separate step for this.

One outstanding question for feedback ->  The configuration tracks the 
parsers created out of the extension, initially.  In the future this list should
have new parsers added to it when they are cloned.  This would mean we 
would want new rest endpoints to edit the configuration ( most likely api to 
add parser to config {name} ) added

This PR is built on top of METRON-777, and should not be merged until 
METRON-777 is in, and this branched is rebased on to master.

### USAGE
For usage, I'll cover creating a parser and deploying it... and deleting it 
using the rest api.

## Preparation
In order to build the parser, we need to build and install the maven plugin 
to build the bundles, as well as the maven archetype itself.
At this time, since they are not published to apache maven, this will be a 
requirement.

- Build the bundle plugin
```
cd bundles-maven-plugin && mvn -q install && cd ..
```
- Build the archetype
```
cd metron-maven-archetypes/metron-maven-parser-extension-archetype
mvn install
cd ../..
```

## Create and build a new parser extension
- Make a directory and cd into it
- Create from archetype
```
mvn archetype:generate -DarchetypeCatalog=local
```
   - choose the org.apache.metron:metron-maven-parser-extension-archetype 
(Apache Maven Parser Extension Archetype for Metron) option
   - Fill out the information (information on the parameters is in the 
README for the archetype)
   - NOTE: at this time, the version must match the metronVersion.  Maven 
is using version in the parent pom chain, and messing up the plugin dependency
- cd into the created directory and build the parser extension
```
mvn package
```

This will result in the tar.gz being created inside the 
{NAME}-parser-assembly project/target directory

## start full_dev
- run vagrant up from metron-deployment/vagrant/full_dev_platform

## add a slot to storm
- log into ambari on http://node1:8080
- add a new slot to the storm config

## Open swagger
- log into ambari on http://node1:8080
- go into the metron service | quicklinks
- start swagger with user | password

## Install your parser
In swagger, using the parser-extension-controller
- drop down the POST method
- use the file selector to select the created .tar.gz
- execute
The extension should now be installed:
- you should be able to use the other GET methods to review the generated 
configuration.
  - the configuration will have the extensionID to use for the other {name} 
operations
- you should be able to view the installed files in 
hdfs/apps/metron/patterns/{parsername}, hdfs/apps/metron/extensions_alt_lib/

## Start the parser using rest
Still in swagger
- Use the Kafka Controller and the Storm Controller to create a 
{parserName} topic, and start a storm job for {parserName}
- In the storm UI you should see the storm instance for the parser
You should be able to verify using the rest api and the various controllers 
that the configurations are there and the topics exist and the storm jobs as 
well


## Uninstall your parser
In swagger, using the parser-extension-controller
- drop down the DELETE method
- fill in the name parameter with the extensionID from the config
- exectute

When complete you should see that the kafka topic, storm job, the various 
configurations are all gone, along with the patterns and other

[GitHub] metron issue #580: METRON-942 [NO MERGE UNTIL METRON-777] Rest api and confi...

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/580
  
TRAVIS!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #580: METRON-942 [NO MERGE UNTIL METRON-777] Rest api an...

Github user ottobackwards closed the pull request at:

https://github.com/apache/metron/pull/580


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...

Github user JonZeolla closed the pull request at:

https://github.com/apache/metron/pull/586


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...

GitHub user JonZeolla reopened a pull request:

https://github.com/apache/metron/pull/586

METRON-508 Expand Elasticsearch templates to support the standard bro logs

## Contributor Comments
This PR makes it easier for someone with an existing bro install to send 
some of their log files into Metron, based off of a combination of the [bro 
documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and 
a fresh install of bro 2.5.  There are future plans to expand on this via 
[METRON-518](https://issues.apache.org/jira/browse/METRON-518) and 
[METRON-908](https://issues.apache.org/jira/browse/METRON-908).  Specifically, 
this attempts to provide initial support the default-on fields of the following 
logs:
 - 
[Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info)
 - 
[DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info)
 - 
[FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info)
 - 
[Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info)
 - 
[CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo)
 - 
[SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info)
 - 
[SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info)
 - 
[Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info)
 - 
[Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info)
 - 
[DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info)
 - 
[SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info)
 - 
[Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info)
 - 
[Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info)
 - 
[X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info)
 - 
[DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo)


## Testing
1.  Create a working directory and pull in this PR
```
mkdir ~/metron-508
git clone https://github.com/apache/metron ~/metron-508/metron
cd ~/metron-508/metron
git remote add jonzeolla https://github.com/jonzeolla/metron
git pull jonzeolla METRON-508
```
1.  Modify 
[this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20)
 to remove `sensors,` (to spin up the real sensors).
```
sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
metron-deployment/vagrant/full-dev-platform/Vagrantfile
```
1.  Start up full-dev.
```
cd metron-deployment/vagrant/full-dev-platform
vagrant up
```
1.  Set up the environment in full-dev.
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin
service monit stop && service sensor-stubs stop bro && broctl stop
yum -y install jq wireshark
```
1.  Configure kafka in local.bro.
```
sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = 
set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, 
SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, 
Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' 
/usr/local/bro/share/bro/site/local.bro
echo "redef Kafka::debug = \"all\";" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Known::cert_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Software::asset_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
sed -i '86 a @load 
policy/protocols/dhcp/known-devices-and-hostnames.bro' 
/usr/local/bro/share/bro/site/local.bro
```
1.  Monitor the bro kafka topic
```
# Open a new terminal
cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
```
1.  Monitor the storm logs.
```
# Open a new terminal
cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
# Look at the storm logs (The "failed to parse" errors for ip_src_addr 
and ip_dst_addr are expected, and should be addressed as a part of METRON-939)
tail

[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/624#discussion_r124811748
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserDriver.java
 ---
@@ -0,0 +1,168 @@
+/**
--- End diff --

How bad would it be to keep both types of tests?  If we actually wanted to 
do the old integration tests for certain types of build ( outside of travis )?  
Like validation of releases or pre-pr commit?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #614: METRON-992: Create performance tuning guide

2017-06-29 Thread dlyle65535

Github user dlyle65535 commented on the issue:

https://github.com/apache/metron/pull/614
  
+1. Used this with a largish instance, worked well, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron