[GitHub] metron issue #905: METRON-1417: Disable pcap-service by default in Monit

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/905
  
Just a heads up my first manual test failed with full dev because of this 
guy:

https://github.com/apache/metron/blob/master/metron-deployment/roles/monit-start/tasks/main.yml#L36
```
- name: Find the services installed on the host
  shell: monit summary | tail -n +3 | awk -F"'" '{print $2}'
  register: installed_services

# which services that need started are actually installed on this host?
- set_fact:
installed_services_to_start: "{{ services_to_start | 
intersect(installed_services.stdout_lines) }}"
```

It grabs all the things in monit and attempts to start them.

I removed the pcapservice monit script since we don't really need it for 
full dev. Full dev still spins up as normal and the pcapservice is no longer 
running.

```
[root@node1 ~]# ps aux|grep pcapservice
root  2167  0.0  0.0 103312   888 pts/0S+   01:00   0:00 grep 
pcapservice

[root@node1 ~]# monit summary
The Monit daemon 5.14 uptime: 2h 31m

Process 'snort' Running
Process 'bro'   Running
System 'node1'  Not monitored
```

This is all set, pending Travis getting fixed.


---

Re: Master is failed in Travis

[Michael Miklavcic]

Yeah, this seems to be breaking in every build at this point. I am going to
look into it tomorrow.

On Mon, Jan 22, 2018 at 8:29 AM, Nick Allen  wrote:

> I had copied the wrong text into the bug.  I fixed that.
>
> On Mon, Jan 22, 2018 at 10:22 AM, Casey Stella  wrote:
>
> > This could be one of those intermittent test failures related to timing.
> > Specifically this:
> >
> > test(org.apache.metron.rest.controller.SensorIndexingConfigController
> > IntegrationTest)
> >  Time elapsed: 0.064 sec  <<< FAILURE!
> > java.lang.AssertionError: Status expected:<404> but was:<200>
> > at org.springframework.test.util.AssertionErrors.fail(
> > AssertionErrors.java:54)
> > at org.springframework.test.util.AssertionErrors.assertEquals(
> > AssertionErrors.java:81)
> > at org.springframework.test.web.servlet.result.
> > StatusResultMatchers$10.match(StatusResultMatchers.java:664)
> > at org.springframework.test.web.servlet.MockMvc$1.andExpect(
> > MockMvc.java:171)
> > at org.apache.metron.rest.controller.
> > SensorIndexingConfigControllerIntegrationTest.test(
> > SensorIndexingConfigControllerIntegrationTest.java:146)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:62)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> > FrameworkMethod.java:47)
> > at org.junit.internal.runners.model.ReflectiveCallable.run(
> > ReflectiveCallable.java:12)
> > at org.junit.runners.model.FrameworkMethod.invokeExplosively(
> > FrameworkMethod.java:44)
> > at org.junit.internal.runners.statements.InvokeMethod.
> > evaluate(InvokeMethod.java:17)
> > at org.junit.internal.runners.statements.RunBefores.
> > evaluate(RunBefores.java:26)
> > at org.springframework.test.context.junit4.statements.
> > RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.
> > java:75)
> > at org.springframework.test.context.junit4.statements.
> > RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.
> java:86)
> > at org.springframework.test.context.junit4.statements.
> > SpringRepeat.evaluate(SpringRepeat.java:84)
> > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> > at org.springframework.test.context.junit4.
> > SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)
> > at org.springframework.test.context.junit4.
> > SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> > at org.junit.runners.ParentRunner$1.schedule(
> ParentRunner.java:63)
> > at org.junit.runners.ParentRunner.runChildren(
> > ParentRunner.java:236)
> > at org.junit.runners.ParentRunner.access$000(
> ParentRunner.java:53)
> > at org.junit.runners.ParentRunner$2.evaluate(
> > ParentRunner.java:229)
> > at org.springframework.test.context.junit4.statements.
> > RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.
> java:61)
> > at org.springframework.test.context.junit4.statements.
> > RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
> > at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> > at org.springframework.test.context.junit4.
> > SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
> > at org.apache.maven.surefire.junit4.JUnit4Provider.execute(
> > JUnit4Provider.java:283)
> > at org.apache.maven.surefire.junit4.JUnit4Provider.
> > executeWithRerun(JUnit4Provider.java:173)
> > at org.apache.maven.surefire.junit4.JUnit4Provider.
> > executeTestSet(JUnit4Provider.java:153)
> > at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(
> > JUnit4Provider.java:128)
> > at org.apache.maven.surefire.booter.ForkedBooter.
> > invokeProviderInSameClassLoader(ForkedBooter.java:203)
> > at org.apache.maven.surefire.booter.ForkedBooter.
> > runSuitesInProcess(ForkedBooter.java:155)
> > at org.apache.maven.surefire.booter.ForkedBooter.main(
> > ForkedBooter.java:103)
> >
> >
> >
> > On Mon, Jan 22, 2018 at 10:21 AM, Nick Allen  wrote:
> >
> > > I had created this JIRA for the specific issue earlier this morning.  I
> > > have no idea why it is breaking and I am not currently looking into it.
> > > Definitely nothing to do with the most recent commit.
> > >
> > > https://issues.apache.org/jira/browse/METRON-1414
> > >
> > >
> > > On Mon, Jan 22, 2018 at 10:18 AM, Otto Fowler  >
> > > wrote:
> > >
> > > > https://travis-ci.org/apache/metron/builds/330900667
> > > >
> 

[GitHub] metron issue #905: METRON-1417: Disable pcap-service by default in Monit

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/905
  
+1, anything to save room on vagrant at the moment.


---

[GitHub] metron pull request #905: METRON-1417: Disable pcap-service by default in Mo...

[mmiklavc]

GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/905

METRON-1417: Disable pcap-service by default in Monit

## Contributor Comments

https://issues.apache.org/jira/browse/METRON-1417

Free up some full dev mem. We don't run the pcap topology atm, so little 
point in running this service by default. Users can manually re-enable

I'm spinning up full dev to finish testing now.

## Testing

Spin up full dev and verify pcapservice is not running
```
ps aux|grep pcapservice
```

That is all. Now enjoy a nap, you've earned it. ;-)

## Pull Request Checklist

In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/metron METRON-1417

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/905.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #905


commit 4a2c9359258d03b7064eff9c52585b3abcc0a0fa
Author: Michael Miklavcic 
Date:   2018-01-23T19:20:10Z

METRON-1417: Disable pcap-service by default in Monit




---

[GitHub] metron issue #904: METRON-1400: Elasticsearch service check fails in Ambari

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/904
  
Travis failure appears unrelated


---

[GitHub] metron issue #904: METRON-1400: Elasticsearch service check fails in Ambari

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/904
  
+1 by inspection, thanks!


---

[GitHub] metron pull request #904: METRON-1400: Elasticsearch service check fails in ...

[mmiklavc]

GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/904

METRON-1400: Elasticsearch service check fails in Ambari

## Contributor Comments
Addresses https://issues.apache.org/jira/browse/METRON-1400

This file hasn't been touched since at least 2016, but the file=sys.stdout 
arg to the Logging statement is causing errors. I removed it and changed the ES 
host:port info to pull from the Ambari configs as we probably wanted all along. 
The regex is similar to what we did to setup links for ES health and indexes 
under ES Ambari Quicklinks.

## Testing

### Basic Test

1. Spin up full dev
2. Open Ambari at http://node1:8080
3. Click on Elasticsearch in the service list
4. Verify ES is running. Start it if it is not running.
5. Click "Service Actions" and choose "Run Service Check" from the dropdown
6. The service check should pass

### Check ES http port

1. Click on Elasticsearch in the service list
2. Select "Configs" in the tabs across the top of the page
3. Enter "http_port" in the filter box
4. Change the default "9200-9300" value to something else, e.g. "9700-9800"
5. Click save and restart ES
6. Once ES comes up again, click "Service Actions" and choose "Run Service 
Check" again from the dropdown
7. The service check should still pass

## Pull Request Checklist

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/metron METRON-1400

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/904.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #904


commit 49aa0bb7c0c6b8ccc5fdbb58592a20c992b4579a
Author: Michael Miklavcic 
Date:   2018-01-23T18:34:34Z

Elasticsearch service check fails in Ambari




---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/853
  
I don't see anything that should be problematic in ES 5.6.2, but can you 
confirm @merrimanr?


---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/903
  
I am continuing a final round of testing to make sure these changes are 
solid.  I wanted to get the PR open so I could start getting feedback as soon 
as possible. 


---

[GitHub] metron pull request #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/903

METRON-1370 Create Full Dev Equivalent for Ubuntu



This PR adds a single node, VM based deployment mechanism based on Ubuntu 
Trusty.  This has feature parity with the existing CentOS single node VM.  This 
will help us continue to improve and support the Ubuntu-based DEB packages that 
were added in #868.

### Changes

* The Metron, Elasticsearch, and Kibana MPacks were updated to support 
installation on Ubuntu.  This included making the following changes.

* Adding repo definitions for Ubuntu
* Changing the default location of the `/etc/sysconfig/elasticsearch` 
configuration to work across platforms
* Adding an additional configuration in the MPack to allow 
Elasticsearch to perform a memory lock on Ubuntu
* Support configuration of Elasticsearch on a platform running Systemd; 
like Ubuntu 

* The Elasticsearch MPack was enhanced to correctly perform a service 
check.  Previously the hostname and port were hard coded and the service check 
was not functional in most environments.

* The Ubuntu DEBs were enhanced to allow for the addition of "maintainer 
scripts".  This allows the Management UI and Alerts UI service scripts to be 
installed with the DEBs.  This was needed to reach feature parity with the RPMs.

* The name `full-dev-platform` no longer makes sense IMHO.  We do not have 
a `quick-dev-platform` any longer to distinguish it from.  Now the main 
distinguisher is the underlying operating system.  I renamed our "Full Dev" 
development environment to `metron-on-centos` and `metron-on-ubuntu`.  I am 
completely open to community suggestions on what these should be named.

```
metron-deployment/vagrant/
├── README.md
├── fastcapa
├── metron-on-centos
└── metron-on-ubuntu
```

* The organization of `metron-deployment` had to change so that assets 
could be reused across both the CentOS and Ubuntu deployments.  All shared 
Ansible assets were moved to `metron-deployment/ansible`.  This includes the 
Ansible modules, playbooks and roles that are all used across multiple 
environments.

```
metron-deployment/ansible/
├── extra_modules
├── playbooks
└── roles
```

* The following Ansible roles had to be updated to support both CentOS and 
Ubuntu based on small differences in package names and conventions.

* `ambari-common` 
* `ambari-config`
* `ambari-master`
* `ambari-slave`
* `libselinux-python`
* `ntp`

* A new role was added to enable swap space; `enable-swap`.  This is 
required for the Ubuntu deployment as the underlying Ubuntu Trusty image does 
not have swap space enabled by default.

* The `metron-builder` role was changed to selectively build either the 
DEBs or the RPMs as needed.

* The `metron-rpms` role was renamed to `metron-packages` and was also 
enhanced to create the local repository on a Ubuntu host.  This also 
selectively copies either RPMs or DEBs to the VM as needed.

* Monit is no longer needed and so has been removed from all VM 
deployments.  Monit was added prior to the existence of our Mpack installer.  
It is no longer needed

* Removed the `metron-streaming` role which is no longer applicable since 
these functions are now performed by the MPack.

### Testing

I have performed the following testing based on these changes.

- [ ] Run Metron on CentOS.  

```
cd metron-deployment/vagrant/metron-on-centos
vagrant up
```

* Validate that alerts are visible within the Alerts UI.
* Validate that the Metron Service Check completes successfully.
* Validate that the Elasticsearch service check completes successfully.

Be sure to shutdown and halt or destroy this VM before moving on to the 
next step.

- [ ] Run Metron on Ubuntu.

```
cd metron-deployment/vagrant/metron-on-centos
vagrant up
```

* Validate that alerts are visible within the Alerts UI.
* Validate that the Metron Service Check completes successfully.
* Validate that the Elasticsearch service check completes successfully.

Be sure to shutdown and halt or destroy this VM before moving on to the 
next step.

- [ ] Run one of the Fastcapa test environments.

```
cd metron-deployment/vagrant/fastcapa/centos-7.1/
vagrant up
```

* If the process fails at the task "fastcapa : Restart for modified 
kernel params" simply run `vagrant provision` again.
* Ensure that the deployment process reports success.  Automated 
validation is 

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
I will address 6 shortly.  For 5, should we explore a more flexible store 
in this PR?  Or at least validate that an RDBMS is the right choice?  I think 
this is something we should tackle now as it will likely affect future work as 
well.  Does it make sense to start a discuss thread on this topic?


---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/853
  
We're good on 1-4.

For point 5.  I'd still like to see a note in the upgrading, even if it 
gets removed when the more flexible store is added. I don't know when we'll get 
around to using a different store, and if it ends up not being this release, 
we'll need to add this anyway.

I think point 6 still needs to be addressed.



---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/853
  
Just ducking in here, @merrimanr is this ready for review?  Specifically, 
the responses to @justinleet 's questions have all been factored into the 
current code for this PR, right?  If yes, @justinleet would you agree or is 
there more work to be done to react to your comments?


---