IMPALA-6070: Expose using Docker to run tests faster. Allows running the tests that make up the "core" suite in about 2 hours. By comparison, https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/buildTimeTrend tends to run in about 3.5 hours.
This commit: * Adds "echo" statements in a few places, to facilitate timing. * Adds --skip-parallel/--skip-serial flags to run-tests.py, and exposes them in run-all-tests.sh. * Marks TestRuntimeFilters as a serial test. This test runs queries that need > 1GB of memory, and, combined with other tests running in parallel, can kill the parallel test suite. * Adds "test-with-docker.py", which runs a full build, data load, and executes tests inside of Docker containers, generating a timeline at the end. In short, one container is used to do the build and data load, and then this container is re-used to run various tests in parallel. All logs are left on the host system. Besides the obvious win of getting test results more quickly, this commit serves as an example of how to get various bits of Impala development working inside of Docker containers. For example, Kudu relies on atomic rename of directories, which isn't available in most Docker filesystems, and entrypoint.sh works around it. In addition, the timeline generated by the build suggests where further optimizations can be made. Most obviously, dataload eats up a precious ~30-50 minutes, on a largely idle machine. This work is significantly CPU and memory hungry. It was developed on a 32-core, 120GB RAM Google Compute Engine machine. I've worked out parallelism configurations such that it runs nicely on 60GB of RAM (c4.8xlarge) and over 100GB (eg., m4.10xlarge, which has 160GB). There is some simple logic to guess at some knobs, and there are knobs. By and large, EC2 and GCE price machines linearly, so, if CPU usage can be kept up, it's not wasteful to run on bigger machines. Change-Id: I82052ef31979564968effef13a3c6af0d5c62767 Reviewed-on: http://gerrit.cloudera.org:8080/9085 Reviewed-by: Philip Zeyliger <phi...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/impala/repo Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/93543cfb Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/93543cfb Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/93543cfb Branch: refs/heads/2.x Commit: 93543cfbca657a2064a86c030a59736f907d60d6 Parents: e380c8f Author: Philip Zeyliger <phi...@cloudera.com> Authored: Sat Oct 21 21:27:00 2017 -0700 Committer: Impala Public Jenkins <impala-public-jenk...@gerrit.cloudera.org> Committed: Wed Apr 11 22:56:00 2018 +0000 ---------------------------------------------------------------------- bin/bootstrap_system.sh | 11 + bin/rat_exclude_files.txt | 1 + bin/run-all-tests.sh | 5 +- buildall.sh | 6 +- docker/README.md | 5 + docker/annotate.py | 34 ++ docker/entrypoint.sh | 329 +++++++++++++++ docker/monitor.py | 329 +++++++++++++++ docker/test-with-docker.py | 579 ++++++++++++++++++++++++++ docker/timeline.html.template | 142 +++++++ testdata/bin/run-all.sh | 2 + tests/query_test/test_runtime_filters.py | 3 + tests/run-tests.py | 34 +- 13 files changed, 1467 insertions(+), 13 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/bin/bootstrap_system.sh ---------------------------------------------------------------------- diff --git a/bin/bootstrap_system.sh b/bin/bootstrap_system.sh index 3a9c42b..3f88dd3 100755 --- a/bin/bootstrap_system.sh +++ b/bin/bootstrap_system.sh @@ -89,10 +89,14 @@ function apt-get { return 1 } +echo ">>> Installing packages" + apt-get update apt-get --yes install apt-utils apt-get --yes install git +echo ">>> Checking out Impala" + # If there is no Impala git repo, get one now if ! [[ -d ~/Impala ]] then @@ -103,6 +107,7 @@ SET_IMPALA_HOME="export IMPALA_HOME=$(pwd)" echo "$SET_IMPALA_HOME" >> ~/.bashrc eval "$SET_IMPALA_HOME" +echo ">>> Installing build tools" apt-get --yes install ccache g++ gcc libffi-dev liblzo2-dev libkrb5-dev \ krb5-admin-server krb5-kdc krb5-user libsasl2-dev libsasl2-modules \ libsasl2-modules-gssapi-mit libssl-dev make maven ninja-build ntp \ @@ -122,6 +127,8 @@ SET_JAVA_HOME="export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" echo "$SET_JAVA_HOME" >> "${IMPALA_HOME}/bin/impala-config-local.sh" eval "$SET_JAVA_HOME" +echo ">>> Configuring system" + sudo service ntp stop sudo ntpdate us.pool.ntp.org # If on EC2, use Amazon's ntp servers @@ -191,10 +198,14 @@ sudo chown $(whoami) /var/lib/hadoop-hdfs/ echo "* - nofile 1048576" | sudo tee -a /etc/security/limits.conf # LZO is not needed to compile or run Impala, but it is needed for the data load +echo ">>> Checking out Impala-lzo" if ! [[ -d ~/Impala-lzo ]] then git clone https://github.com/cloudera/impala-lzo.git ~/Impala-lzo fi + +echo ">>> Checking out and building hadoop-lzo" + if ! [[ -d ~/hadoop-lzo ]] then git clone https://github.com/cloudera/hadoop-lzo.git ~/hadoop-lzo http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/bin/rat_exclude_files.txt ---------------------------------------------------------------------- diff --git a/bin/rat_exclude_files.txt b/bin/rat_exclude_files.txt index 5bb13f0..a2b6267 100644 --- a/bin/rat_exclude_files.txt +++ b/bin/rat_exclude_files.txt @@ -79,6 +79,7 @@ tests/comparison/ORACLE.txt bin/distcc/README.md tests/comparison/POSTGRES.txt docs/README.md +docker/README.md be/src/thirdparty/pcg-cpp-0.98/README.md # http://www.apache.org/legal/src-headers.html: "Test data for which the addition of a http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/bin/run-all-tests.sh ---------------------------------------------------------------------- diff --git a/bin/run-all-tests.sh b/bin/run-all-tests.sh index 4ca0f54..4743a4f 100755 --- a/bin/run-all-tests.sh +++ b/bin/run-all-tests.sh @@ -55,6 +55,8 @@ fi # Extra arguments passed to start-impala-cluster for tests. These do not apply to custom # cluster tests. : ${TEST_START_CLUSTER_ARGS:=} +# Extra args to pass to run-tests.py +: ${RUN_TESTS_ARGS:=} if [[ "${TARGET_FILESYSTEM}" == "local" ]]; then # TODO: Remove abort_on_config_error flag from here and create-load-data.sh once # checkConfiguration() accepts the local filesystem (see IMPALA-1850). @@ -188,7 +190,8 @@ do if [[ "$EE_TEST" == true ]]; then # Run end-to-end tests. # KERBEROS TODO - this will need to deal with ${KERB_ARGS} - if ! "${IMPALA_HOME}/tests/run-tests.py" ${COMMON_PYTEST_ARGS} ${EE_TEST_FILES}; then + if ! "${IMPALA_HOME}/tests/run-tests.py" ${COMMON_PYTEST_ARGS} \ + ${RUN_TESTS_ARGS} ${EE_TEST_FILES}; then #${KERB_ARGS}; TEST_RET_CODE=1 fi http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/buildall.sh ---------------------------------------------------------------------- diff --git a/buildall.sh b/buildall.sh index 9d8b15e..56cdb9a 100755 --- a/buildall.sh +++ b/buildall.sh @@ -334,7 +334,7 @@ bootstrap_dependencies() { echo "SKIP_TOOLCHAIN_BOOTSTRAP is true, skipping download of Python dependencies." echo "SKIP_TOOLCHAIN_BOOTSTRAP is true, skipping toolchain bootstrap." else - echo "Downloading Python dependencies" + echo ">>> Downloading Python dependencies" # Download all the Python dependencies we need before doing anything # of substance. Does not re-download anything that is already present. if ! "$IMPALA_HOME/infra/python/deps/download_requirements"; then @@ -344,7 +344,7 @@ bootstrap_dependencies() { echo "Finished downloading Python dependencies" fi - echo "Downloading and extracting toolchain dependencies." + echo ">>> Downloading and extracting toolchain dependencies." "$IMPALA_HOME/bin/bootstrap_toolchain.py" echo "Toolchain bootstrap complete." fi @@ -357,6 +357,7 @@ build_fe() { # Build all components. build_all_components() { + echo ">>> Building all components" # Build the Impala frontend, backend and external data source API. MAKE_IMPALA_ARGS+=" -fe -cscope -tarballs" if [[ -e "$IMPALA_LZO" ]] @@ -421,6 +422,7 @@ start_test_cluster_dependencies() { # This does all data loading, except for the metastore snapshot which must be loaded # earlier before the cluster is running. load_test_data() { + echo ">>> Loading test data" "$IMPALA_HOME/bin/create_testdata.sh" # We have 4 cases: # - test-warehouse and metastore snapshots exists. http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/docker/README.md ---------------------------------------------------------------------- diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 0000000..f4e7709 --- /dev/null +++ b/docker/README.md @@ -0,0 +1,5 @@ +# Docker-related scripts for Impala + +`test-with-docker.py` runs the Impala build and tests inside of Docker +containers, parallelizing the test execution across test suites. See that file +for more details. http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/docker/annotate.py ---------------------------------------------------------------------- diff --git a/docker/annotate.py b/docker/annotate.py new file mode 100755 index 0000000..f83854f --- /dev/null +++ b/docker/annotate.py @@ -0,0 +1,34 @@ +#!/usr/bin/python -u +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Prepends input with timestamp. Unlike similar perl version, +# produces microsecond timestamps. Python is unavailable in +# the base Ubuntu image. +# +# Note that "python -u" disables buffering. + +import sys +import datetime + +while True: + line = sys.stdin.readline() + if line == "": + sys.exit(0) + sys.stdout.write(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f ")) + sys.stdout.write(line) http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/docker/entrypoint.sh ---------------------------------------------------------------------- diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh new file mode 100755 index 0000000..d371d25 --- /dev/null +++ b/docker/entrypoint.sh @@ -0,0 +1,329 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Entrypoint code for test-with-docker.py containers. test-with-docker.py +# will create Docker containers with this script as the entrypoint, +# with a variety of arguments. See test-with-docker.py for a more +# general overview. +# +# This assumes that the following are already mounted inside +# the container: +# /etc/localtime -> /mnt/localtime +# Helps timestamps be in the time zone of the host +# $IMPALA_HOME [a git repo of Impala] -> /repo +# Used to check out Impala afresh +# $IMPALA_HOME/logs/docker/<n1>/<n2> -> /logs +# Used to save logs out to host. +# <n1> represents the --name passed into +# test-with-docker for the test run. <n2> +# indicates which specific container is being run. +# ~/.ccache [configurable] -> /ccache +# Used to speed up builds. +# +# Usage: +# entrypoint.sh build <uid> +# entrypoint.sh test_suite <suite> +# where <suite> is one of: BE_TEST JDBC_TEST CLUSTER_TEST +# EE_TEST_SERIAL EE_TEST_PARALLEL + +# Boostraps the container by creating a user and adding basic tools like Python and git. +# Takes a uid as an argument for the user to be created. +function build() { + # Handy for testing. + if [[ $TEST_TEST_WITH_DOCKER ]]; then + # We sleep busily so that CPU metrics will show usage, to + # better exercise the timeline code. + echo sleeping busily for 4 seconds + bash -c 'while [[ $SECONDS -lt 4 ]]; do :; done' + return + fi + + # Configure timezone, so any timestamps that appear are coherent with the host. + configure_timezone + + # Assert we're superuser. + [ "$(id -u)" = 0 ] + if id $1 2> /dev/null; then + echo "User with id $1 already exists. Please run this as a user id missing from " \ + "the base Ubuntu container." + echo + echo "Container users:" + paste <(cut -d : -f3 /etc/passwd) <(cut -d : -f1 /etc/passwd) | sort -n + exit 1 + fi + apt-get update + apt-get install -y sudo git lsb-release python + + adduser --disabled-password --gecos "" --uid $1 impdev + echo "impdev ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers + + su impdev -c "$0 build_impdev" +} + +# Sets up Impala environment +function impala_environment() { + pushd /home/impdev/Impala + export IMPALA_HOME=/home/impdev/Impala + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + source bin/impala-config.sh + popd +} + +# Starts SSH and PostgreSQL; configures container as necessary; +# prepares Kudu for starting. +function boot_container() { + pushd /home/impdev/Impala + + # Required for metastore + sudo service postgresql start + + # Required for starting HBase + sudo service ssh start + + # Make log directories. This is typically done in buildall.sh. + mkdir -p logs/be_tests logs/fe_tests/coverage logs/ee_tests logs/custom_cluster_tests + + # Update /etc/hosts to remove the entry for the unique docker hostname, + # and instead point it to 127.0.0.1. Otherwise, HttpFS returns Location: + # redirects to said hostname, but the relevant datanode isn't listening + # on the wildcard address. + sed -e /$(hostname)/d /etc/hosts -e /127.0.0.1/s,localhost,"localhost $(hostname)," \ + > /tmp/hosts + # "sed -i" in place doesn't work on Docker, because /etc/hosts is a bind mount. + sudo cp /tmp/hosts /etc/hosts + + echo Hostname: $(hostname) + echo Hosts file: + cat /etc/hosts + + # Make a copy of Kudu's WALs to avoid isue with Docker filesystems (aufs and + # overlayfs) that don't support os.rename(2) on directories, which Kudu + # requires. We make a fresh copy of the data, in which case rename(2) works + # presumably because there's only one layer involved. See + # https://issues.apache.org/jira/browse/KUDU-1419. + cd /home/impdev/Impala/testdata + for x in cluster/cdh*/node-*/var/lib/kudu/*/wal; do + mv $x $x-orig + cp -r $x-orig $x + rm -r $x-orig + done + + # Wait for postgresql to really start; if it doesn't, Hive Metastore will fail to start. + for i in {1..120}; do + echo connecting to postgresql attempt $i + if sudo -u postgres psql -c "select 1"; then + break + else + sleep 2 + fi + done + sudo -u postgres psql -c "select 1" + + popd +} + +# Runs bootstrap_system.sh and then builds Impala. +function build_impdev() { + # Assert we're impdev now. + [ "$(id -un)" = impdev ] + + # Link in ccache from host. + ln -s /ccache /home/impdev/.ccache + + # Instead of doing a full "git clone" of /repo, which is the host's checkout, + # we only fetch one branch, without tags. This keeps the checkout + # considerably lighter. + mkdir /home/impdev/Impala + pushd /home/impdev/Impala + git init + git fetch /repo --no-tags HEAD + git checkout -b test-with-docker FETCH_HEAD + + # Link in logs. Logs are on the host since that's the most important thing to + # look at after the tests are run. + ln -sf /logs logs + + bin/bootstrap_system.sh + impala_environment + + # Builds Impala and loads test data. + # Note that IMPALA-6494 prevents us from using shared library builds, + # which are smaller and thereby speed things up. + ./buildall.sh -noclean -format -testdata -skiptests + + # Shut down things cleanly. + testdata/bin/kill-all.sh + + # Shutting down PostgreSQL nicely speeds up it's start time for new containers. + sudo service postgresql stop + + # Clean up things we don't need to reduce image size + find be -name '*.o' -execdir rm '{}' + # ~1.6GB + + popd +} + +# Runs a suite passed in as the first argument. Tightly +# coupled with Impala's run-all-tests and the suite names. +# from test-with-docker.py. +# +# Before running tests, starts up the minicluster. +function test_suite() { + cd /home/impdev/Impala + + # These test suites are for testing. + if [[ $1 == NOOP ]]; then + return 0 + fi + if [[ $1 == NOOP_FAIL ]]; then + return 1 + fi + if [[ $1 == NOOP_SLEEP_FOREVER ]]; then + # Handy to test timeouts. + while true; do sleep 60; done + fi + + # Assert that we're running as impdev + [ "$(id -un)" = impdev ] + + # Assert that /home/impdev/Impala/logs is a symlink to /logs. + [ "$(readlink /home/impdev/Impala/logs)" = /logs ] + + boot_container + impala_environment + + # By default, the JVM will use 1/4 of your OS memory for its heap size. For a + # long-running test, this will delay GC inside of impalad's leading to + # unnecessarily large process RSS footprints. We cap the heap size at + # a more reasonable size. Note that "test_insert_large_string" fails + # at 2g and 3g, so the suite that includes it (EE_TEST_PARALLEL) gets + # additional memory. + # + # Similarly, bin/start-impala-cluster typically configures the memlimit + # to be 80% of the machine memory, divided by the number of daemons. + # If multiple containers are to be run simultaneously, this is scaled + # down in test-with-docker.py (and further configurable with --impalad-mem-limit-bytes) + # and passed in via $IMPALAD_MEM_LIMIT_BYTES to the container. There is a + # relationship between the number of parallel tests that can be run by py.test and this + # limit. + JVM_HEAP_GB=2 + if [[ $1 = EE_TEST_PARALLEL ]]; then + JVM_HEAP_GB=4 + fi + export TEST_START_CLUSTER_ARGS="--jvm_args=-Xmx${JVM_HEAP_GB}g \ + --impalad_args=--mem_limit=$IMPALAD_MEM_LIMIT_BYTES" + + # BE tests don't require the minicluster, so we can run them directly. + if [[ $1 = BE_TEST ]]; then + # IMPALA-6494: thrift-server-test fails in Ubuntu16.04 for the moment; skip it. + export SKIP_BE_TEST_PATTERN='thrift-server-test*' + if ! bin/run-backend-tests.sh; then + echo "Tests $1 failed!" + return 1 + else + echo "Tests $1 succeeded!" + return 0 + fi + fi + + # Start the minicluster + testdata/bin/run-all.sh + + export MAX_PYTEST_FAILURES=0 + # Choose which suite to run; this is how run-all.sh chooses between them. + export FE_TEST=false + export BE_TEST=false + export EE_TEST=false + export JDBC_TEST=false + export CLUSTER_TEST=false + + eval "export ${1}=true" + + if [[ ${1} = "EE_TEST_SERIAL" ]]; then + # We bucket the stress tests with the parallel tests. + export RUN_TESTS_ARGS="--skip-parallel --skip-stress" + export EE_TEST=true + elif [[ ${1} = "EE_TEST_PARALLEL" ]]; then + export RUN_TESTS_ARGS="--skip-serial" + export EE_TEST=true + fi + + ret=0 + + # Run tests. + if ! time -p bin/run-all-tests.sh; then + ret=1 + echo "Tests $1 failed!" + else + echo "Tests $1 succeeded!" + fi + # Oddly, I've observed bash fail to exit (and wind down the container), + # leading to test-with-docker.py hitting a timeout. Killing the minicluster + # daemons fixes this. + testdata/bin/kill-all.sh || true + return $ret +} + +# Ubuntu's tzdata package is very finnicky, and if you +# mount /etc/localtime from the host to the container directly, +# it fails to install. However, if you make it a symlink +# and configure /etc/timezone to something that's not an +# empty string, you'll get the right behavior. +# +# The post installation script is findable by looking for "tzdata.postinst" +# +# Use this command to reproduce the Ubuntu issue: +# docker run -v /etc/localtime:/mnt/localtime -ti ubuntu:16.04 bash -c ' +# date +# ln -sf /mnt/localtime /etc/localtime +# date +%Z > /etc/timezone +# date +# apt-get update > /dev/null +# apt-get install tzdata +# date' +function configure_timezone() { + if ! diff -q /etc/localtime /mnt/localtime 2> /dev/null; then + ln -sf /mnt/localtime /etc/localtime + date +%Z > /etc/timezone + fi +} + +function main() { + set -e + + # Run given command + CMD="$1" + shift + + echo ">>> ${CMD} $@ (begin)" + set -x + if "${CMD}" "$@"; then + ret=0 + else + ret=$? + fi + set +x + echo ">>> ${CMD} $@ ($ret) (end)" + exit $ret +} + +# Run main() unless we're being sourced. +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + main "$@" +fi http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/docker/monitor.py ---------------------------------------------------------------------- diff --git a/docker/monitor.py b/docker/monitor.py new file mode 100644 index 0000000..64cde0c --- /dev/null +++ b/docker/monitor.py @@ -0,0 +1,329 @@ +#!/usr/bin/python +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Monitors Docker containers for CPU and memory usage, and +# prepares an HTML timeline based on said monitoring. +# +# Usage example: +# mon = monitor.ContainerMonitor("monitoring.txt") +# mon.start() +# # container1 is an object with attributes id, name, and logfile. +# mon.add(container1) +# mon.add(container2) +# mon.stop() +# timeline = monitor.Timeline("monitoring.txt", +# [container1, container2], +# re.compile(">>> ")) +# timeline.create("output.html") + +import datetime +import json +import logging +import os +import shutil +import subprocess +import threading +import time + + +# Unit for reporting user/system CPU seconds in cpuacct.stat. +# See https://www.kernel.org/doc/Documentation/cgroup-v1/cpuacct.txt and time(7). +USER_HZ = os.sysconf(os.sysconf_names['SC_CLK_TCK']) + + +def total_memory(): + """Returns total RAM on system, in GB.""" + return _memory()[0] + + +def used_memory(): + """Returns total used RAM on system, in GB.""" + return _memory()[1] + + +def _memory(): + """Returns (total, used) memory on system, in GB. + + Used is computed as total - available. + + Calls "free" and parses output. Sample output for reference: + + total used free shared buffers cache available + Mem: 126747197440 26363965440 56618553344 31678464 2091614208 41673064448 99384889344 + Swap: 0 0 0 + """ + + free_lines = subprocess.check_output(["free", "-b", "-w"]).split('\n') + free_grid = [x.split() for x in free_lines] + # Identify columns for "total" and "available" + total_idx = free_grid[0].index("total") + available_idx = free_grid[0].index("available") + total = int(free_grid[1][1 + total_idx]) + available = int(free_grid[1][1 + available_idx]) + used = total - available + total_gb = total / (1024.0 * 1024.0 * 1024.0) + used_gb = used / (1024.0 * 1024.0 * 1024.0) + return (total_gb, used_gb) + + +def datetime_to_seconds_since_epoch(dt): + """Converts a Python datetime to seconds since the epoch.""" + return time.mktime(dt.timetuple()) + + +def split_timestamp(line): + """Parses timestamp at beginning of a line. + + Returns a tuple of seconds since the epoch and the rest + of the line. Returns None on parse failures. + """ + LENGTH = 26 + FORMAT = "%Y-%m-%d %H:%M:%S.%f" + t = line[:LENGTH] + return (datetime_to_seconds_since_epoch(datetime.datetime.strptime(t, FORMAT)), + line[LENGTH + 1:]) + + +class ContainerMonitor(object): + """Monitors Docker containers. + + Monitoring data is written to a file. An example is: + + 2018-02-02 09:01:37.143591 d8f640989524be3939a70557a7bf7c015ba62ea5a105a64c94472d4ebca93c50 cpu user 2 system 5 + 2018-02-02 09:01:37.143591 d8f640989524be3939a70557a7bf7c015ba62ea5a105a64c94472d4ebca93c50 memory cache 11481088 rss 4009984 rss_huge 0 mapped_file 8605696 dirty 24576 writeback 0 pgpgin 4406 pgpgout 624 pgfault 3739 pgmajfault 99 inactive_anon 0 active_anon 3891200 inactive_file 7614464 active_file 3747840 unevictable 0 hierarchical_memory_limit 9223372036854771712 total_cache 11481088 total_rss 4009984 total_rss_huge 0 total_mapped_file 8605696 total_dirty 24576 total_writeback 0 total_pgpgin 4406 total_pgpgout 624 total_pgfault 3739 total_pgmajfault 99 total_inactive_anon 0 total_active_anon 3891200 total_inactive_file 7614464 total_active_file 3747840 total_unevictable 0 + + That is, the format is: + + <timestamp> <container> cpu user <usercpu> system <systemcpu> + <timestamp> <container> memory <contents of memory.stat without newlines> + + <usercpu> and <systemcpu> are in the units of USER_HZ. + See https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt for documentation + on memory.stat; it's in the "memory" cgroup, often mounted at + /sys/fs/cgroup/memory/<cgroup>/memory.stat. + + This format is parsed back by the Timeline class below and should + not be considered an API. + """ + + def __init__(self, output_path, frequency_seconds=1): + """frequency_seconds is how often metrics are gathered""" + self.containers = [] + self.output_path = output_path + self.keep_monitoring = None + self.monitor_thread = None + self.frequency_seconds = frequency_seconds + + def start(self): + self.keep_monitoring = True + self.monitor_thread = threading.Thread(target=self._monitor) + self.monitor_thread.setDaemon(True) + self.monitor_thread.start() + + def stop(self): + self.keep_monitoring = False + self.monitor_thread.join() + + def add(self, container): + """Adds monitoring for container, which is an object with property 'id'.""" + self.containers.append(container) + + @staticmethod + def _metrics_from_stat_file(root, container, stat): + """Returns metrics stat file contents. + + root: a cgroups root (a path as a string) + container: an object with string attribute id + stat: a string filename + + Returns contents of <root>/<container.id>/<stat> + with newlines replaced with spaces. + Returns None on errors. + """ + dirname = os.path.join(root, "docker", container.id) + if not os.path.isdir(dirname): + # Container may no longer exist. + return None + try: + statcontents = file(os.path.join(dirname, stat)).read() + return statcontents.replace("\n", " ").strip() + except IOError, e: + # Ignore errors; cgroup can disappear on us. + logging.warning("Ignoring exception reading cgroup. " + + "This can happen if container just exited. " + str(e)) + return None + + def _monitor(self): + """Monitors CPU usage of containers. + + Otput is stored in self.output_path. + Also, keeps track of minimum and maximum memory usage (for the machine). + """ + # Ubuntu systems typically mount cpuacct cgroup in /sys/fs/cgroup/cpu,cpuacct, + # but this can vary by OS distribution. + all_cgroups = subprocess.check_output( + "findmnt -n -o TARGET -t cgroup --source cgroup".split() + ).split("\n") + cpuacct_root = [c for c in all_cgroups if "cpuacct" in c][0] + memory_root = [c for c in all_cgroups if "memory" in c][0] + logging.info("Using cgroups: cpuacct %s, memory %s", cpuacct_root, memory_root) + self.min_memory_usage_gb = None + self.max_memory_usage_gb = None + + with file(self.output_path, "w") as output: + while self.keep_monitoring: + # Use a single timestamp for a given round of monitoring. + now = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f") + for c in self.containers: + cpu = self._metrics_from_stat_file(cpuacct_root, c, "cpuacct.stat") + memory = self._metrics_from_stat_file(memory_root, c, "memory.stat") + if cpu: + output.write("%s %s cpu %s\n" % (now, c.id, cpu)) + if memory: + output.write("%s %s memory %s\n" % (now, c.id, memory)) + output.flush() + + # Machine-wide memory usage + m = used_memory() + if self.min_memory_usage_gb is None: + self.min_memory_usage_gb, self.max_memory_usage_gb = m, m + else: + self.min_memory_usage_gb = min(self.min_memory_usage_gb, m) + self.max_memory_usage_gb = max(self.max_memory_usage_gb, m) + time.sleep(self.frequency_seconds) + + +class Timeline(object): + """Given metric and log data for containers, creates a timeline report. + + This is a standalone HTML file with a timeline for the log files and CPU charts for + the containers. The HTML uses https://developers.google.com/chart/ for rendering + the charts, which happens in the browser. + """ + + def __init__(self, monitor_file, containers, interesting_re): + self.monitor_file = monitor_file + self.containers = containers + self.interesting_re = interesting_re + + def logfile_timeline(self, container): + """Returns a list of (name, timestamp, line) tuples for interesting lines in + the container's logfile. container is expected to have name and logfile attributes. + """ + interesting_lines = [ + line.strip() + for line in file(container.logfile) + if self.interesting_re.search(line)] + return [(container.name,) + split_timestamp(line) for line in interesting_lines] + + @staticmethod + def parse_metrics(f): + """Parses timestamped metric lines. + + Given metrics lines like: + + 2017-10-25 10:08:30.961510 87d5562a5fe0ea075ebb2efb0300d10d23bfa474645bb464d222976ed872df2a cpu user 33 system 15 + + Returns an iterable of (ts, container, user_cpu, system_cpu) + """ + prev_by_container = {} + for line in f: + ts, rest = split_timestamp(line.rstrip()) + try: + container, metric_type, rest2 = rest.split(" ", 2) + if metric_type != "cpu": + continue + _, user_cpu_s, _, system_cpu_s = rest2.split(" ", 3) + except: + logging.warning("Skipping metric line: %s", line) + continue + + prev_ts, prev_user, prev_system = prev_by_container.get( + container, (None, None, None)) + user_cpu = int(user_cpu_s) + system_cpu = int(system_cpu_s) + if prev_ts is not None: + # Timestamps are seconds since the epoch and are floats. + dt = ts - prev_ts + assert type(dt) == float + if dt != 0: + yield ts, container, (user_cpu - prev_user)/dt/USER_HZ,\ + (system_cpu - prev_system)/dt/USER_HZ + prev_by_container[container] = ts, user_cpu, system_cpu + + def create(self, output): + # Read logfiles + timelines = [] + for c in self.containers: + if not os.path.exists(c.logfile): + logging.warning("Missing log file: %s", c.logfile) + continue + timelines.append(self.logfile_timeline(c)) + + # Convert timelines to JSON + min_ts = None + timeline_json = [] + for timeline in timelines: + for current_line, next_line in zip(timeline, timeline[1:]): + name, ts_current, msg = current_line + _, ts_next, _ = next_line + timeline_json.append( + [name, msg, ts_current, ts_next] + ) + if not timeline_json: + logging.warning("No timeline data; skipping timeline") + return + + min_ts = min(x[2] for x in timeline_json) + + for row in timeline_json: + row[2] = row[2] - min_ts + row[3] = row[3] - min_ts + + # metrics_by_container: container -> [ ts, user, system ] + metrics_by_container = dict() + max_metric_ts = 0 + container_by_id = dict() + for c in self.containers: + container_by_id[c.id] = c + + for ts, container_id, user, system in self.parse_metrics(file(self.monitor_file)): + container = container_by_id.get(container_id) + if not container: + continue + + if ts > max_metric_ts: + max_metric_ts = ts + if ts < min_ts: + # We ignore metrics that show up before the timeline's + # first messages. This largely avoids a bug in the + # Google Charts visualization code wherein one of the series seems + # to wrap around. + continue + metrics_by_container.setdefault( + container.name, []).append((ts - min_ts, user, system)) + + with file(output, "w") as o: + template_path = os.path.join(os.path.dirname(__file__), "timeline.html.template") + shutil.copyfileobj(file(template_path), o) + o.write("\n<script>\nvar data = \n") + json.dump(dict(timeline=timeline_json, metrics=metrics_by_container, + max_ts=(max_metric_ts - min_ts)), o, indent=2) + o.write("</script>") + o.close() http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/docker/test-with-docker.py ---------------------------------------------------------------------- diff --git a/docker/test-with-docker.py b/docker/test-with-docker.py new file mode 100755 index 0000000..59640b4 --- /dev/null +++ b/docker/test-with-docker.py @@ -0,0 +1,579 @@ +#!/usr/bin/python +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +CLI_HELP = """\ +Runs tests inside of docker containers, parallelizing different types of +tests. This script first creates a docker container, checks out this repo +into it, bootstraps the container with Impala dependencies, and builds Impala +and its test data. Then, it saves the resulting container, and launches new +containers to run tests in parallel. An HTML, visual timeline is generated +as part of the build, in logs/docker/*/timeline.html. +""" + +# To execute run: +# docker/test-with-docker.py +# After waiting for that to finish, inspect results in logs/docker. +# +# Visually, the timeline looks as follows, produced on a 32-core, 100GB RAM +# machine: +# ....... 1h05m Checkout, setup machine, build (8m with ccache), +# generate testdata (52m); missing ccache +# adds about 7 minutes (very sensitive to number +# of available cores) +# ... 11m Commit the Docker container +# . 10m FE tests +# . 10m JDBC tests +# .... 45m serial EE tests +# ...... 1h02m cluster tests +# ... 31m BE (C++) tests +# .... 36m parallel EE tests +# Total time: 2h25m. +# +# CPU usage is sustained high for the parallel EE tests and for +# the C++ compile (when it's not ccache'd), but is otherwise low. +# Because every parallel track consumes memory (a cluster), +# increasing parallelism and memory must be balanced. +# +# Memory usage is thorny. The minicluster memory can +# be tweaked somewhat by how much memory to give to the JVM +# and what to set --mem_limit too. Furthermore, parallel +# cluster tests use more memory when more parallelism happens. +# +# The code that runs inside of the containers is in entrypoint.sh, +# whereas the code that invokes docker is here. +# +# We avoid using Dockerfile and "docker build": they make it hard or impossible +# to cross-mount host directories into containers or use --privileged, and using +# them would require generating them dynamically. They're more trouble than +# they're worth for this use case. +# +# In practice, the containers are about 100GB (with 45GB +# being test data and ~40GB being the tests). +# +# Requirements: +# * Docker +# This has been tested on Ubuntu16.04 with Docker +# from the Ubuntu repos, i.e., Docker 1.13.1. +# * About 150 GB of disk space available to Docker. +# * 75GB of RAM. +# +# This script tries to clean up images and containers created by this process, though +# this can be disabled for debugging. +# +# To clean up containers and images manually, you can use: +# for x in $(docker ps -aq --filter label=pwd=$IMPALA_HOME); do +# docker stop $x; docker rm $x; done +# for x in $(docker images -q --filter label=pwd=$IMPALA_HOME); do docker rmi $x; done +# +# Core dumps: +# On an Ubuntu host, core dumps and Docker don't mix by default, because apport is not +# running inside of the container. See https://github.com/moby/moby/issues/11740 +# To enable core dumps, run the following command on the host: +# $echo 'core.%e.%p' | sudo tee /proc/sys/kernel/core_pattern +# +# TODOs: +# - Support for executing other flavors, like exhaustive, or file systems, +# like S3. +# +# Suggested speed improvement TODOs: +# - Speed up testdata generation +# - Skip generating test data for variants not being run +# - Make container image smaller; perhaps make BE test binaries +# smaller +# - Split up cluster tests into two groups +# - Analyze .xml junit files to find slow tests; eradicate +# or move to different suite. +# - Avoid building BE tests, and build them during execution, +# saving on container space as well as baseline build +# time. + +# We do not use Impala's python environment here, nor do we depend on +# non-standard python libraries to avoid needing extra build steps before +# triggering this. +import argparse +import datetime +import logging +import multiprocessing +import multiprocessing.pool +import os +import re +import subprocess +import sys +import tempfile +import time + +if __name__ == '__main__' and __package__ is None: + sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + import monitor + +base = os.path.dirname(os.path.abspath(__file__)) + + +def main(): + + logging.basicConfig(level=logging.INFO, + format='%(asctime)s %(threadName)s: %(message)s') + + default_parallel_test_concurrency, default_suite_concurrency, default_memlimit_gb = \ + _compute_defaults() + parser = argparse.ArgumentParser( + description=CLI_HELP, formatter_class=argparse.ArgumentDefaultsHelpFormatter) + group = parser.add_mutually_exclusive_group() + group.add_argument('--cleanup-containers', dest="cleanup_containers", + action='store_true', default=True, + help='Removes containers when finished.') + group.add_argument('--no-cleanup-containers', + dest="cleanup_containers", action='store_false') + group = parser.add_mutually_exclusive_group() + parser.add_argument( + '--parallel-test-concurrency', type=int, + default=default_parallel_test_concurrency, + help='For the ee-test-parallel suite, how many tests to run concurrently.') + parser.add_argument( + '--suite-concurrency', type=int, default=default_suite_concurrency, + help='Number of concurrent suites to run in parallel.') + parser.add_argument( + '--impalad-mem-limit-bytes', type=int, default=default_memlimit_gb, + help='Memlimit to pass to impalad for miniclusters.') + group.add_argument( + '--cleanup-image', dest="cleanup_image", + action='store_true', default=True, + help="Whether to remove image when done.") + group.add_argument('--no-cleanup-image', dest="cleanup_image", action='store_false') + parser.add_argument( + '--build-image', metavar='IMAGE', + help='Skip building, and run tests on pre-existing image.') + parser.add_argument( + '--suite', metavar='VARIANT', action='append', + help="Run specific test suites; can be specified multiple times. \ + If not specified, all tests are run. Choices: " + ",".join(ALL_SUITES)) + parser.add_argument( + '--name', metavar='NAME', + help="Use a specific name for the test run. The name is used " + + "as a prefix for the container and image names, and " + + "as part of the log directory naming. Defaults to include a timestamp.", + default=datetime.datetime.now().strftime("i-%Y%m%d-%H%M%S")) + parser.add_argument('--timeout', metavar='MINUTES', + help="Timeout for test suites, in minutes.", + type=int, + default=60*2) + parser.add_argument('--ccache-dir', metavar='DIR', + help="CCache directory to use", + default=os.path.expanduser("~/.ccache")) + parser.add_argument('--test', action="store_true") + args = parser.parse_args() + + if not args.suite: + args.suite = ALL_SUITES + t = TestWithDocker( + build_image=args.build_image, suites=args.suite, + name=args.name, timeout=args.timeout, cleanup_containers=args.cleanup_containers, + cleanup_image=args.cleanup_image, ccache_dir=args.ccache_dir, test_mode=args.test, + parallel_test_concurrency=args.parallel_test_concurrency, + suite_concurrency=args.suite_concurrency, + impalad_mem_limit_bytes=args.impalad_mem_limit_bytes) + + logging.getLogger('').addHandler( + logging.FileHandler(os.path.join(_make_dir_if_not_exist(t.log_dir), "log.txt"))) + + logging.info("Arguments: %s", args) + + ret = t.run() + t.create_timeline() + + if not ret: + sys.exit(1) + + +def _compute_defaults(): + """Compute default config options based on memory. + + The goal is to work reasonably on machines with + about 60GB of memory, like Amazon's c4.8xlarge (36 CPUs, 60GB) + or c5.9xlarge (36 CPUs, 72GB) or m4.4xlarge (16 CPUs, 64 GB). + + Based on some experiments, we set up defaults for different + machine sizes based on memory, with an eye towards + having reasonable runtimes as well. + + Experiments on memory usage: + + suite parallelism usage + Xmx memlimit + ee-test-parallel 4GB 8 5GB 33GB + ee-test-parallel 4GB 16 7GB 37GB + ee-test-serial 4GB - 5GB 18GB + cluster-test 4GB - - 13GB + be-test 4GB - 10GB 19GB + fe-test 4GB - 10GB 9GB + """ + total_memory_gb = monitor.total_memory() + cpus = multiprocessing.cpu_count() + logging.info("CPUs: %s Memory (GB): %s", cpus, total_memory_gb) + + parallel_test_concurrency = min(cpus, 8) + memlimit_gb = 7 + + if total_memory_gb >= 95: + suite_concurrency = 4 + parallel_test_concurrency = min(cpus, 12) + elif total_memory_gb >= 65: + suite_concurrency = 3 + elif total_memory_gb >= 35: + suite_concurrency = 2 + else: + logging.warning("This tool should be run on a machine with more memory.") + suite_concurrency = 1 + + return parallel_test_concurrency, suite_concurrency, memlimit_gb * 1024 * 1024 * 1024 + + +# The names of all the test tracks supported. NOOP isn't included here, but is +# handy for testing. These are organized slowest-to-fastest, so that, when +# parallelism of suites is limited, the total time is not impacted. +ALL_SUITES = [ + "EE_TEST_SERIAL", + "EE_TEST_PARALLEL", + "CLUSTER_TEST", + "BE_TEST", + "FE_TEST", + "JDBC_TEST", +] + + +def _call(args, check=True): + """Wrapper for calling a subprocess. + + args is the first argument of subprocess.Popen, typically + an array, e.g., ["echo", "hi"]. + + If check is set, raise an exception on failure. + """ + logging.info("Calling: %s", args) + if check: + subprocess.check_call(args, stdin=None) + else: + return subprocess.call(args, stdin=None) + + +def _check_output(*args, **kwargs): + """Wrapper for subprocess.check_output, with logging.""" + logging.info("Running: %s, %s; cmdline: %s.", args, kwargs, " ".join(*args)) + return subprocess.check_output(*args, **kwargs) + + +def _make_dir_if_not_exist(*parts): + d = os.path.join(*parts) + if not os.path.exists(d): + os.makedirs(d) + return d + + +class Container(object): + """Encapsulates a container, with some metadata.""" + + def __init__(self, id_, name, logfile, exitcode=None, running=None): + self.id = id_ + self.name = name + self.logfile = logfile + self.exitcode = exitcode + self.running = running + self.start = None + self.end = None + + def runtime_seconds(self): + if self.start and self.end: + return self.end - self.start + + def __str__(self): + return "Container<" + \ + ",".join(["%s=%s" % (k, v) for k, v in self.__dict__.items()]) \ + + ">" + + +class TestWithDocker(object): + """Tests Impala using Docker containers for parallelism.""" + + def __init__(self, build_image, suites, name, timeout, cleanup_containers, + cleanup_image, ccache_dir, test_mode, + suite_concurrency, parallel_test_concurrency, + impalad_mem_limit_bytes): + self.build_image = build_image + self.suites = [TestSuiteRunner(self, suite) for suite in suites] + self.name = name + self.containers = [] + self.timeout_minutes = timeout + self.git_root = _check_output(["git", "rev-parse", "--show-toplevel"]).strip() + self.cleanup_containers = cleanup_containers + self.cleanup_image = cleanup_image + self.image = None + if build_image and cleanup_image: + # Refuse to clean up external image. + raise Exception("cleanup_image and build_image cannot be both specified") + self.ccache_dir = ccache_dir + self.log_dir = os.path.join(self.git_root, "logs", "docker", self.name) + self.monitoring_output_file = os.path.join(self.log_dir, "metrics.txt") + self.monitor = monitor.ContainerMonitor(self.monitoring_output_file) + self.test_mode = test_mode + self.suite_concurrency = suite_concurrency + self.parallel_test_concurrency = parallel_test_concurrency + self.impalad_mem_limit_bytes = impalad_mem_limit_bytes + + def _create_container(self, image, name, logdir, logname, entrypoint, extras=None): + """Returns a new container. + + logdir - subdirectory to create under self.log_dir, + which will get mounted to /logs + logname - name of file in logdir that will be created + extras - extra arguments to pass to docker + entrypoint - entrypoint arguments, as a list. + """ + if extras is None: + extras = [] + if self.test_mode: + extras = ["-e", "TEST_TEST_WITH_DOCKER=true"] + extras + + container_id = _check_output([ + "docker", "create", + # Required for some of the ntp handling in bootstrap and Kudu; + # requirement may be lifted in newer Docker versions. + "--privileged", + "--name", name, + "--hostname", name, + # Label with the git root directory for easier cleanup + "--label=pwd=" + self.git_root, + # Consistent locales + "-e", "LC_ALL=C", + "-e", "IMPALAD_MEM_LIMIT_BYTES=" + + str(self.impalad_mem_limit_bytes), + # Mount the git directory so that clones can be local + "-v", self.git_root + ":/repo:ro", + "-v", self.ccache_dir + ":/ccache", + # Share timezone between host and container + "-v", "/etc/localtime:/mnt/localtime", + "-v", _make_dir_if_not_exist(self.log_dir, + logdir) + ":/logs", + "-v", base + ":/mnt/base:ro"] + + extras + + [image] + + entrypoint).strip() + return Container(name=name, id_=container_id, + logfile=os.path.join(self.log_dir, logdir, logname)) + + def _run_container(self, container): + """Runs container, and returns True if the container had a successful exit value. + + This blocks while the container is running. The container output is + run through annotate.py to add timestamps and saved into the container's log file. + """ + container.running = True + + with file(container.logfile, "w") as log_output: + container.start = time.time() + # Sets up a "docker start ... | annotate.py > logfile" pipeline using + # subprocess. + annotate = subprocess.Popen( + [os.path.join(self.git_root, "docker", "annotate.py")], + stdin=subprocess.PIPE, + stdout=log_output, + stderr=log_output) + + logging.info("Starting container %s; logging to %s", container.name, + container.logfile) + docker = subprocess.Popen(["docker", "start", "--attach", container.id], + stdin=None, stdout=annotate.stdin, stderr=annotate.stdin) + + ret = docker.wait() + annotate.stdin.close() + annotate.wait() + + logging.info("Container %s returned %s", container, ret) + container.exitcode = ret + container.running = False + container.end = time.time() + return ret == 0 + + @staticmethod + def _stop_container(container): + """Stops container. Ignores errors (e.g., if it's already exited).""" + _call(["docker", "stop", container.id], check=False) + if container.running: + container.end = time.time() + container.running = False + + @staticmethod + def _rm_container(container): + """Removes container.""" + _call(["docker", "rm", container.id], check=False) + + def _create_build_image(self): + """Creates the "build image", with Impala compiled and data loaded.""" + container = self._create_container( + image="ubuntu:16.04", name=self.name, + logdir="build", + logname="log-build.txt", + # entrypoint.sh will create a user with our uid; this + # allows the shared file systems to work seamlessly + entrypoint=["/mnt/base/entrypoint.sh", "build", str(os.getuid())]) + self.containers.append(container) + self.monitor.add(container) + try: + logging.info("Docker container for build: %s", container) + _check_output(["docker", "start", container.id]) + if not self._run_container(container): + raise Exception("Build container failed.") + logging.info("Committing docker container.") + self.image = _check_output( + ["docker", "commit", + "-c", "LABEL pwd=" + self.git_root, + container.id, "impala:built-" + self.name]).strip() + logging.info("Committed docker image: %s", self.image) + finally: + if self.cleanup_containers: + self._stop_container(container) + self._rm_container(container) + + def _run_tests(self): + start_time = time.time() + timeout_seconds = self.timeout_minutes * 60 + deadline = start_time + timeout_seconds + pool = multiprocessing.pool.ThreadPool(processes=self.suite_concurrency) + outstanding_suites = [] + for suite in self.suites: + suite.task = pool.apply_async(suite.run) + outstanding_suites.append(suite) + + ret = True + while time.time() < deadline and len(outstanding_suites) > 0: + for suite in list(outstanding_suites): + task = suite.task + if task.ready(): + this_task_ret = task.get() + outstanding_suites.remove(suite) + if this_task_ret: + logging.info("Suite %s succeeded.", suite.name) + else: + logging.info("Suite %s failed.", suite.name) + ret = False + time.sleep(10) + if len(outstanding_suites) > 0: + for container in self.containers: + self._stop_container(container) + for suite in outstanding_suites: + suite.task.get() + raise Exception("Tasks not finished within timeout (%s minutes): %s" % + (self.timeout_minutes, ",".join([ + suite.name for suite in outstanding_suites]))) + return ret + + def run(self): + # Create logs directories and ccache dir. + _make_dir_if_not_exist(self.ccache_dir) + _make_dir_if_not_exist(self.log_dir) + + self.monitor.start() + try: + if not self.build_image: + self._create_build_image() + else: + self.image = self.build_image + ret = self._run_tests() + logging.info("Containers:") + for c in self.containers: + def to_success_string(exitcode): + if exitcode == 0: + return "SUCCESS" + return "FAILURE" + logging.info("%s %s %s %s", to_success_string(c.exitcode), c.name, c.logfile, + c.runtime_seconds()) + return ret + finally: + self.monitor.stop() + if self.cleanup_image and self.image: + _call(["docker", "rmi", self.image], check=False) + logging.info("Memory usage: %s GB min, %s GB max", + self.monitor.min_memory_usage_gb, + self.monitor.max_memory_usage_gb) + + # Strings (really, regular expressions) pulled out into to the visual timeline. + _INTERESTING_STRINGS = [ + ">>> ", + ] + _INTERESTING_RE = re.compile("|".join("(%s)" % (s,) for s in _INTERESTING_STRINGS)) + + def create_timeline(self): + """Creates timeline into log directory.""" + timeline = monitor.Timeline( + monitor_file=self.monitoring_output_file, + containers=self.containers, + interesting_re=self._INTERESTING_RE) + timeline.create(os.path.join(self.log_dir, "timeline.html")) + + +class TestSuiteRunner(object): + """Runs a single test suite.""" + + def __init__(self, test_with_docker, suite): + self.test_with_docker = test_with_docker + self.suite = suite + self.task = None + self.name = self.suite.lower() + + def run(self): + """Runs given test. Returns true on success, based on exit code.""" + test_with_docker = self.test_with_docker + suite = self.suite + self.start = time.time() + + # io-file-mgr-test expects a real-ish file system at /tmp; + # we mount a temporary directory into the container to appease it. + tmpdir = tempfile.mkdtemp(prefix=test_with_docker.name + "-" + self.name) + # Container names are sometimes used as hostnames, and DNS names shouldn't + # have underscores. + container_name = test_with_docker.name + "-" + self.name.replace("_", "-") + + container = test_with_docker._create_container( + image=test_with_docker.image, + name=container_name, + extras=[ + "-v", tmpdir + ":/tmp", + "-u", str(os.getuid()), + "-e", "NUM_CONCURRENT_TESTS=" + + str(test_with_docker.parallel_test_concurrency), + ], + logdir=self.name, + logname="log-test-" + self.suite + ".txt", + entrypoint=["/mnt/base/entrypoint.sh", "test_suite", suite]) + + test_with_docker.containers.append(container) + test_with_docker.monitor.add(container) + try: + return test_with_docker._run_container(container) + except: + return False + finally: + logging.info("Cleaning up containers for %s" % (suite,)) + test_with_docker._stop_container(container) + if test_with_docker.cleanup_containers: + test_with_docker._rm_container(container) + + +if __name__ == "__main__": + main() http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/docker/timeline.html.template ---------------------------------------------------------------------- diff --git a/docker/timeline.html.template b/docker/timeline.html.template new file mode 100644 index 0000000..c8de821 --- /dev/null +++ b/docker/timeline.html.template @@ -0,0 +1,142 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +<!-- + +Template/header for a timeline visualization of a multi-container build. +The timelines represent interesting log lines, with one row per container. +The charts represent CPU usage within those containers. + +To use this, concatenate this with a '<script>' block defining +a global variable named data. + +The expected format of data is exemplified by the following, +and is tightly coupled with the implementation generating +it in monitor.py. The intention of this unfriendly file format +is to do as much munging as plausible in Python. + +To make the visualization relative to the start time (i.e., to say that all +builds start at 00:00), the timestamps are all seconds since the build began. +To make the visualization work with them, the timestamps are then converted +into local time, and get displayed reasonably. This is a workaround to the fact +that the visualization library for the timelines does not accept any data types +that represent duration, but we still want timestamp-style formatting. + +var data = { + // max timestamp seen, in seconds since the epoch + "max_ts": 8153.0, + // map of container name to an array of metrics + "metrics": { + "i-20180312-140548-ee-test-serial": [ + // a single metric point is an array of timestamp, user CPU, system CPU + // CPU is the percent of 1 CPU used since the previous timestamp. + [ + 4572.0, + 0.11, + 0.07 + ] + ] + }, + // Array of timelines + "timeline": [ + // a timeline entry contains a name (for the entire row of the timeline), + // the message (for a segment of the timeline), and start and end timestamps + // for the segment. + [ + "i-20180312-140548", + "+ echo '>>> build' '4266 (begin)'", + 0.0, + 0.0 + ] + ] +} +--> + +<script type="text/javascript" src="https://www.gstatic.com/charts/loader.js";></script> + +<script type="text/javascript"> +google.charts.load("current", {packages:["timeline", "corechart"]}); +google.charts.setOnLoadCallback(drawChart); + +function ts_to_hms(secs) { + var s = secs % 60; + var m = Math.floor(secs / 60) % 60; + var h = Math.floor(secs / (60 * 60)); + return [h, m, s]; +} + +/* Returns a Date object corresponding to secs seconds since the epoch, in + * localtime. Date(x) and Date(0, 0, 0, 0, 0, 0, 0, x) differ in that the + * former returns UTC whereas the latter returns the browser local time. + * For consistent handling within this visualization, we use localtime. + * + * Beware that local time can be discontinuous around time changes. + */ +function ts_to_date(secs) { + // secs may be a float, so we use millis as a common denominator unit + var millis = 1000 * secs; + return new Date(1970 /* yr; beginning of unix epoch */, 0 /* mo */, 0 /* d */, + 0 /* hr */, 0 /* min */, 0 /* sec */, millis); +} + +function drawChart() { + var container = document.getElementById('container'); + var timelineContainer = document.createElement("div"); + container.appendChild(timelineContainer); + var chart = new google.visualization.Timeline(timelineContainer); + var dataTable = new google.visualization.DataTable(); + dataTable.addColumn({ type: 'string', id: 'Position' }); + dataTable.addColumn({ type: 'string', id: 'Name' }); + // timeofday isn't supported here + dataTable.addColumn({ type: 'datetime', id: 'Start' }); + dataTable.addColumn({ type: 'datetime', id: 'End' }); + // Timeline + for (i = 0; i < data.timeline.length; ++i) { + var row = data.timeline[i]; + dataTable.addRow([ row[0], row[1], ts_to_date(row[2]), ts_to_date(row[3]) ]); + } + chart.draw(dataTable, { height: "400px" } ); + + for (const k of Object.keys(data.metrics)) { + var lineChart = document.createElement("div"); + container.appendChild(lineChart); + + var dataTable = new google.visualization.DataTable(); + dataTable.addColumn({ type: 'timeofday', id: 'Time' }); + dataTable.addColumn({ type: 'number', id: 'User' }); + dataTable.addColumn({ type: 'number', id: 'System' }); + + for (const row of data.metrics[k]) { + dataTable.addRow([ ts_to_hms(row[0]), row[1], row[2] ]); + } + var options = { + title: 'CPU', + legend: { position: 'bottom' }, + hAxis: { + minValue: [0, 0, 0], + maxValue: ts_to_hms(data.max_ts) + } + }; + + var chart = new google.visualization.LineChart(lineChart); + chart.draw(dataTable, options); + } +} +</script> +<div id="container" style="height: 200px;"></div> http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/testdata/bin/run-all.sh ---------------------------------------------------------------------- diff --git a/testdata/bin/run-all.sh b/testdata/bin/run-all.sh index fb85811..f722b89 100755 --- a/testdata/bin/run-all.sh +++ b/testdata/bin/run-all.sh @@ -35,6 +35,8 @@ fi # Kill and clean data for a clean start. echo "Killing running services..." +# Create log dir, in case there's nothing to kill. +mkdir -p ${IMPALA_CLUSTER_LOGS_DIR} $IMPALA_HOME/testdata/bin/kill-all.sh &>${IMPALA_CLUSTER_LOGS_DIR}/kill-all.log echo "Starting cluster services..." http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/tests/query_test/test_runtime_filters.py ---------------------------------------------------------------------- diff --git a/tests/query_test/test_runtime_filters.py b/tests/query_test/test_runtime_filters.py index c3763ff..14f5884 100644 --- a/tests/query_test/test_runtime_filters.py +++ b/tests/query_test/test_runtime_filters.py @@ -26,6 +26,9 @@ from tests.common.skip import SkipIfLocal WAIT_TIME_MS = specific_build_type_timeout(60000, slow_build_timeout=100000) +# Some of the queries in runtime_filters consume a lot of memory, leading to +# significant memory reservations in parallel tests. +@pytest.mark.execute_serially @SkipIfLocal.multiple_impalad class TestRuntimeFilters(ImpalaTestSuite): @classmethod http://git-wip-us.apache.org/repos/asf/impala/blob/93543cfb/tests/run-tests.py ---------------------------------------------------------------------- diff --git a/tests/run-tests.py b/tests/run-tests.py index 9552d0d..95e0d11 100755 --- a/tests/run-tests.py +++ b/tests/run-tests.py @@ -18,8 +18,10 @@ # under the License. # # Runs the Impala query tests, first executing the tests that cannot be run in parallel -# and then executing the remaining tests in parallel. All additional command line options -# are passed to py.test. +# (the serial tests), then executing the stress tests, and then +# executing the remaining tests in parallel. To run only some of +# these, use --skip-serial, --skip-stress, or --skip-parallel. +# All additional command line options are passed to py.test. from tests.common.impala_cluster import ImpalaCluster from tests.common.impala_service import ImpaladService import itertools @@ -219,6 +221,15 @@ def print_metrics(substring): if __name__ == "__main__": exit_on_error = '-x' in sys.argv or '--exitfirst' in sys.argv + skip_serial = '--skip-serial' in sys.argv + if skip_serial: + sys.argv.remove("--skip-serial") + skip_stress = '--skip-stress' in sys.argv + if skip_stress: + sys.argv.remove("--skip-stress") + skip_parallel = '--skip-parallel' in sys.argv + if skip_parallel: + sys.argv.remove("--skip-parallel") test_executor = TestExecutor(exit_on_error=exit_on_error) # If the user is just asking for --help, just print the help test and then exit. @@ -241,18 +252,21 @@ if __name__ == "__main__": else: print_metrics('connections') # First run query tests that need to be executed serially - base_args = ['-m', 'execute_serially'] - test_executor.run_tests(base_args + build_test_args('serial')) - print_metrics('connections') + if not skip_serial: + base_args = ['-m', 'execute_serially'] + test_executor.run_tests(base_args + build_test_args('serial')) + print_metrics('connections') # Run the stress tests tests - base_args = ['-m', 'stress', '-n', NUM_STRESS_CLIENTS] - test_executor.run_tests(base_args + build_test_args('stress')) - print_metrics('connections') + if not skip_stress: + base_args = ['-m', 'stress', '-n', NUM_STRESS_CLIENTS] + test_executor.run_tests(base_args + build_test_args('stress')) + print_metrics('connections') # Run the remaining query tests in parallel - base_args = ['-m', 'not execute_serially and not stress', '-n', NUM_CONCURRENT_TESTS] - test_executor.run_tests(base_args + build_test_args('parallel')) + if not skip_parallel: + base_args = ['-m', 'not execute_serially and not stress', '-n', NUM_CONCURRENT_TESTS] + test_executor.run_tests(base_args + build_test_args('parallel')) # The total number of tests executed at this point is expected to be >0 # If it is < 0 then the script needs to exit with a non-zero
