GitHub user merrimanr reopened a pull request: https://github.com/apache/metron/pull/858
METRON-1344: Externalize the infrastructural components using integration tests ## Contributor Comments This PR will add infrastructure to our Travis build that will allow the Alerts UI e2e test to be run. There are several outstanding issues that still need to be worked out so DO NOT MERGE this yet. This is a first pass at a potential Docker-based solution and is meant to be a POC. The intention is to facilitate further discussion around the general approach and provide a working example that we can build off of. A good place to start is the .travis file. This provides a good guide on how this infrastructure is spun up. It is assumed Docker and Docker Compose are installed. To use outside of travis in a local dev environment (assuming Mac OSX): 1. Build Metron with `mvn clean install -DskipTests` 1. Create a Docker machine with `metron-contrib/metron-docker/scripts/create-docker-machine.sh` 1. Set the Docker env variables with `eval $(docker-machine env metron-machine)` 1. Build the base Metron image that installs Java: `docker build ./metron-centos/ -t "metron-centos"` 1. Spin up the environment: `cd metron-contrib/metron-docker-e2e/compose && docker-compose up -d` A working environment should now be available. To verify get the Docker machine address with `echo $DOCKER_HOST`. Services should be available on the ports specified in `metron-contrib/metron-docker-e2e/compose/docker-compose.yml`. For example, assuming my $DOCKER_HOST is "tcp://192.168.99.100:2376", Elasticsearch should be available at http://192.168.99.100:9210. At this point only a single e2e test is run due to the significant refactoring being done in https://github.com/apache/metron/pull/857. Here are my thoughts so far based on work in this PR: - The Docker environment creation and startup adds about 3 minutes to the build. This is about what I expected and hopefully we can get this time back as we move other tests to a reusable environment. - I'm not convinced spinning up containers for the Alerts UI and REST are necessary or desired. We may be able to cut some time off the build by running them directly on the Travis host instead of in Docker - I experimented with caching the Docker images but pulling them every time was actually faster There is still significant work to be done including: - I was able to get the full suite of tests to run successfully in previous commits but have all but e2e tests commented out for now to make it easier to see what's going on in Travis. These will need to be added back once we get closer to a final solution. - Once the e2e test refactoring is done those changes will need merged in and tested - The license check is commented out right now because I inadvertently added some new dependency versions (have no idea why). Still need to track that down. - There is an intermittent error that happens when starting up REST. Still working on tracking this down. Looking forward to some feedback. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` #### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron e2e-in-travis Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/858.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #858 ---- commit b8fb307748c4e83e88cd9c156464eb99b5b13025 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T19:05:14Z initial commit commit faabe6789cb83abe6d5ba6005fa10f73225e253a Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T19:08:32Z travis test commit c61ca37491225df77c14363c499c7465bca15651 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T19:09:55Z travis test commit dda0d7a6d7264dbfcbfb8bdc6e6bf2fc3325ecdb Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T19:15:31Z travis test commit 083f4290b8a3129e27bcca15879bf3889361357a Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T19:19:20Z travis test commit 2b4a90e777756d24d1e74ec80cfbf74832f8d648 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T19:56:57Z travis test commit 7d9b47f5de63ceec66891eb4ee1e5122f908a2a9 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T20:18:32Z travis test commit 594d94c754ca6fc0e62a3fa82505598b0e71d7ae Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T20:38:49Z travis test commit cd06db23494f084abb05a3adbbb2b3170927aaf8 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T21:11:01Z travis test commit 75a8d13c4ecc07a69443aaa61d4ba4859f860946 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T21:17:22Z travis test commit cbf2da7597514a3c8e02309f463d8726d84c13be Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T21:49:07Z travis test commit 280fdf5bca481049ae0f83c5deae1f93611959fa Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T22:25:20Z travis test commit 0ef3f2a79824942edace28b57bf93975760177f6 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T22:31:03Z travis test commit b21ce8288bbbabd75e4eb2e877de20db81fe3b8d Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T22:36:16Z travis test commit 4e741903281875f31ccee0c5d90868a329ff2a67 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T22:40:23Z travis test commit 2e6d74ac80ccc40f3f96e29640d5e3b07eba870d Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T22:47:12Z travis test commit f3cb688563fc201ea4178a45369864184a176dec Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T22:51:06Z travis test commit 55ca067f3768f88f7dade0602a00af92f48e7616 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T22:56:04Z travis test commit a3c65db085744fc3b65491211a769cf19a0695e9 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T23:15:29Z travis test commit 4751848e7cf5221976e7bae23cea8fc9ee75a72f Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T23:18:16Z travis test commit 86a0a5db301b0246ae650d67e1e45c4a442ec074 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T23:27:23Z travis test commit ce2c75acbff2486f028543ef2a56ee3f21a58fc0 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T23:36:12Z travis test commit b5c5a94ef53bc154019c623d7a8368ff99950f9d Author: merrimanr <merrim...@gmail.com> Date: 2017-12-01T23:48:56Z travis test commit ce2bf87199bfc5fb2db1c7d1df7c07ff992b8905 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-02T00:04:45Z travis test commit b426a28cc49e21db4907db5c35268c193e69a8da Author: merrimanr <merrim...@gmail.com> Date: 2017-12-02T00:49:24Z travis test commit 09aa8f080c6ccf3c4e9d5fb267fa86f113ee286c Author: merrimanr <merrim...@gmail.com> Date: 2017-12-02T01:22:07Z travis test commit 1e89a1dceea81c25411f596e29cb73964a408b17 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-02T01:23:36Z travis test commit 02c147bcc9543f3bc07c91e06dc11c9b35b3e422 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-02T15:15:21Z travis test commit a693cf95398e8300900530b085c4724b8a822026 Author: merrimanr <merrim...@gmail.com> Date: 2017-12-02T15:21:58Z travis test commit 39d1d96b497b67b40e1aabed23c84a81ec3f009d Author: merrimanr <merrim...@gmail.com> Date: 2017-12-02T15:54:40Z travis test ---- ---