Philip Zeyliger has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9085
Change subject: IMPALA-6070: Expose using Docker to run tests faster. ...................................................................... IMPALA-6070: Expose using Docker to run tests faster. Allows running the tests that make up the "core" suite in about 2 hours. By comparison, https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/buildTimeTrend tends to run in about 3.5 hours. This commit: * Adds "echo" statements in a few places, to facilitate timing. * Adds --skip-parallel/--skip-serial flags to run-tests.py, and exposes them in run-all-tests.sh. * Adds "test-with-docker.py", which runs a full build, data load, and executes tests inside of Docker containers, generating a timeline at the end. In short, one container is used to do the build and data load, and then this container is re-used to run various tests in parallel. All logs are left on the host system. Besides the obvious win of getting test results more quickly, this commit serves as an example of how to get various bits of Impala development working inside of Docker containers. For example, Kudu relies on atomic rename of directories, which isn't available in most Docker filesystems, and entrypoint.sh works around it. In addition, the timeline generated by the build suggests where further optimizations can be made. Most obviously, dataload eats up a precious ~30-50 minutes, on a largely idle machine. This work is significantly CPU and memory hungry. It was developed on a 32-core, 120GB RAM Google Compute Engine machine, and I know that 75GB of those are currently used. By and large, EC2 and GCE price machines linearly, so, if CPU usage can be kept up, it's not wasteful to run on bigger machines. I've run into some fragility with the tests. Specifically, thrift-server-test fails consistently, as it does on my development Ubuntu 16.04 machine as well. Change-Id: I82052ef31979564968effef13a3c6af0d5c62767 --- M bin/bootstrap_system.sh M bin/run-all-tests.sh M buildall.sh A docker/annotate.pl A docker/annotate.py A docker/entrypoint.sh A docker/test-with-docker.py M testdata/bin/run-all.sh M tests/run-tests.py 9 files changed, 1,013 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/9085/1 -- To view, visit http://gerrit.cloudera.org:8080/9085 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I82052ef31979564968effef13a3c6af0d5c62767 Gerrit-Change-Number: 9085 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger <phi...@cloudera.com> Gerrit-Reviewer: Philip Zeyliger <phi...@cloudera.com>
