GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/858
METRON-1344: Externalize the infrastructural components using integration
tests
## Contributor Comments
This PR will add infrastructure to our Travis build that will allow the
Alerts UI e2e test to be run. There are several outstanding issues that still
need to be worked out so DO NOT MERGE this yet.
This is a first pass at a potential Docker-based solution and is meant to
be a POC. The intention is to facilitate further discussion around the general
approach and provide a working example that we can build off of.
A good place to start is the .travis file. This provides a good guide on
how this infrastructure is spun up. It is assumed Docker and Docker Compose
are installed. To use outside of travis in a local dev environment (assuming
Mac OSX):
1. Build Metron with `mvn clean install -DskipTests`
1. Create a Docker machine with
`metron-contrib/metron-docker/scripts/create-docker-machine.sh`
1. Set the Docker env variables with `eval $(docker-machine env
metron-machine)`
1. Build the base Metron image that installs Java: `docker build
./metron-centos/ -t "metron-centos"`
1. Spin up the environment: `cd metron-contrib/metron-docker-e2e/compose
&& docker-compose up -d`
A working environment should now be available. To verify get the Docker
machine address with `echo $DOCKER_HOST`. Services should be available on the
ports specified in
`metron-contrib/metron-docker-e2e/compose/docker-compose.yml`. For example,
assuming my $DOCKER_HOST is "tcp://192.168.99.100:2376", Elasticsearch should
be available at http://192.168.99.100:9210.
At this point only a single e2e test is run due to the significant
refactoring being done in https://github.com/apache/metron/pull/857.
Here are my thoughts so far based on work in this PR:
- The Docker environment creation and startup adds about 3 minutes to the
build. This is about what I expected and hopefully we can get this time back
as we move other tests to a reusable environment.
- I'm not convinced spinning up containers for the Alerts UI and REST are
necessary or desired. We may be able to cut some time off the build by running
them directly on the Travis host instead of in Docker
- I experimented with caching the Docker images but pulling them every time
was actually faster
There is still significant work to be done including:
- I was able to get the full suite of tests to run successfully in previous
commits but have all but e2e tests commented out for now to make it easier to
see what's going on in Travis. These will need to be added back once we get
closer to a final solution.
- Once the e2e test refactoring is done those changes will need merged in
and tested
- The license check is commented out right now because I inadvertently
added some new dependency versions (have no idea why). Still need to track
that down.
- There is an intermittent error that happens when starting up REST. Still
working on tracking this down.
Looking forward to some feedback.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron e2e-in-travis
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/858.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #858
----
commit b8fb307748c4e83e88cd9c156464eb99b5b13025
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T19:05:14Z
initial commit
commit faabe6789cb83abe6d5ba6005fa10f73225e253a
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T19:08:32Z
travis test
commit c61ca37491225df77c14363c499c7465bca15651
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T19:09:55Z
travis test
commit dda0d7a6d7264dbfcbfb8bdc6e6bf2fc3325ecdb
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T19:15:31Z
travis test
commit 083f4290b8a3129e27bcca15879bf3889361357a
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T19:19:20Z
travis test
commit 2b4a90e777756d24d1e74ec80cfbf74832f8d648
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T19:56:57Z
travis test
commit 7d9b47f5de63ceec66891eb4ee1e5122f908a2a9
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T20:18:32Z
travis test
commit 594d94c754ca6fc0e62a3fa82505598b0e71d7ae
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T20:38:49Z
travis test
commit cd06db23494f084abb05a3adbbb2b3170927aaf8
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T21:11:01Z
travis test
commit 75a8d13c4ecc07a69443aaa61d4ba4859f860946
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T21:17:22Z
travis test
commit cbf2da7597514a3c8e02309f463d8726d84c13be
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T21:49:07Z
travis test
commit 280fdf5bca481049ae0f83c5deae1f93611959fa
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T22:25:20Z
travis test
commit 0ef3f2a79824942edace28b57bf93975760177f6
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T22:31:03Z
travis test
commit b21ce8288bbbabd75e4eb2e877de20db81fe3b8d
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T22:36:16Z
travis test
commit 4e741903281875f31ccee0c5d90868a329ff2a67
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T22:40:23Z
travis test
commit 2e6d74ac80ccc40f3f96e29640d5e3b07eba870d
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T22:47:12Z
travis test
commit f3cb688563fc201ea4178a45369864184a176dec
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T22:51:06Z
travis test
commit 55ca067f3768f88f7dade0602a00af92f48e7616
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T22:56:04Z
travis test
commit a3c65db085744fc3b65491211a769cf19a0695e9
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T23:15:29Z
travis test
commit 4751848e7cf5221976e7bae23cea8fc9ee75a72f
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T23:18:16Z
travis test
commit 86a0a5db301b0246ae650d67e1e45c4a442ec074
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T23:27:23Z
travis test
commit ce2c75acbff2486f028543ef2a56ee3f21a58fc0
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T23:36:12Z
travis test
commit b5c5a94ef53bc154019c623d7a8368ff99950f9d
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-01T23:48:56Z
travis test
commit ce2bf87199bfc5fb2db1c7d1df7c07ff992b8905
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-02T00:04:45Z
travis test
commit b426a28cc49e21db4907db5c35268c193e69a8da
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-02T00:49:24Z
travis test
commit 09aa8f080c6ccf3c4e9d5fb267fa86f113ee286c
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-02T01:22:07Z
travis test
commit 1e89a1dceea81c25411f596e29cb73964a408b17
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-02T01:23:36Z
travis test
commit 02c147bcc9543f3bc07c91e06dc11c9b35b3e422
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-02T15:15:21Z
travis test
commit a693cf95398e8300900530b085c4724b8a822026
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-02T15:21:58Z
travis test
commit 39d1d96b497b67b40e1aabed23c84a81ec3f009d
Author: merrimanr <merrim...@gmail.com>
Date: 2017-12-02T15:54:40Z
travis test
----
---
