This is an automated email from the ASF dual-hosted git repository.
mck pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git
The following commit(s) were added to refs/heads/trunk by this push:
new b2d679b Add ubuntu2004_* testing docker files, and deploy them to the
apache organisation in dockerhub
b2d679b is described below
commit b2d679b60ac55f9bfd759fb0376a9e4e4aa3bfd4
Author: Mick Semb Wever <[email protected]>
AuthorDate: Tue Jan 5 22:16:16 2021 +0100
Add ubuntu2004_* testing docker files, and deploy them to the apache
organisation in dockerhub
Also remove ubuntu18* docker files
patch by Mick Semb Wever; reviewed by Sam Tunnicliffe for CASSANDRA-16373
---
docker/testing/README.md | 14 +--
.../testing/ubuntu1810_j11_w_dependencies.docker | 37 -------
docker/testing/ubuntu18_j11.docker | 112 ---------------------
docker/testing/ubuntu1910_j11.docker | 3 +-
.../testing/ubuntu1910_j11_w_dependencies.docker | 4 +-
...ubuntu1810_j11.docker => ubuntu2004_j11.docker} | 37 +++++--
...docker => ubuntu2004_j11_w_dependencies.docker} | 6 +-
jenkins-dsl/cassandra_job_dsl_seed.groovy | 2 +-
8 files changed, 44 insertions(+), 171 deletions(-)
diff --git a/docker/testing/README.md b/docker/testing/README.md
index 8864391..195f95e 100644
--- a/docker/testing/README.md
+++ b/docker/testing/README.md
@@ -1,6 +1,6 @@
# Docker CI Testing
-Docker files in this directory are used to build images used by CircleCI.
These are directly referenced in the `circle.yml` after publishing to
dockerhub. There are two types of images:
+Docker files in this directory are used to build images used by
ci-cassandra.apache.org and CircleCI. These are directly referenced in the
`cassandra_job_dsl_seed.groovy` and `.circleci/config*.yml` files, after
publishing to dockerhub. There are two types of images:
* Base image for Linux distribution to use for testing (e.g.
`ubuntu1910_j11.docker`)
* Caching image that contains git sources, maven and ccm dependencies
@@ -11,11 +11,11 @@ Build images from the parent directory using the following
commands. Change tag
Base image:
-`docker build -t nastra/cassandra-testing-ubuntu1910-java11:20200603 -t
nastra/cassandra-testing-ubuntu1910-java11:latest -f ubuntu1910_j11.docker .`
+`docker build -t apache/cassandra-testing-ubuntu2004-java11:$(date +"%Y%m%d")
-t apache/cassandra-testing-ubuntu2004-java11:latest -f ubuntu2004_j11.docker .`
Caching image:
-`docker build --no-cache -t
nastra/cassandra-testing-ubuntu1910-java11-w-dependencies:20200603 -t
nastra/cassandra-testing-ubuntu1910-java11-w-dependencies:latest -f
ubuntu1910_j11_w_dependencies.docker .`
+`docker build --no-cache -t
apache/cassandra-testing-ubuntu2004-java11-w-dependencies:$(date +"%Y%m%d") -t
apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest -f
ubuntu2004_j11_w_dependencies.docker .`
Please make sure to always tag also by date, so we can go back to that version
in case anything breaks after the next update!
@@ -26,10 +26,12 @@ We are using Docker Hub for storing published images. See
[Quick Start Guide](ht
Push both image references:
```
-docker push nastra/cassandra-testing-ubuntu1910-java11-w-dependencies:20200603
-docker push nastra/cassandra-testing-ubuntu1910-java11-w-dependencies:latest
+docker push apache/cassandra-testing-ubuntu2004-java11:$(date +"%Y%m%d")
+docker push apache/cassandra-testing-ubuntu2004-java11:latest
+docker push apache/cassandra-testing-ubuntu2004-java11-w-dependencies:$(date
+"%Y%m%d")
+docker push apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest
```
## Updating circleci.yml
-Check if the correct image is used in `.circleci/config-2_1.yml` by looking
for the `- image:` value. It should either be set to the date dervived tag
created above, or `:latest`.
+Check if the correct image is used in `cassandra_job_dsl_seed.groovy` and
`.circleci/config*.yml`. It should either be set to the date dervived tag
created above, or `:latest`.
diff --git a/docker/testing/ubuntu1810_j11_w_dependencies.docker
b/docker/testing/ubuntu1810_j11_w_dependencies.docker
deleted file mode 100644
index 8f25165..0000000
--- a/docker/testing/ubuntu1810_j11_w_dependencies.docker
+++ /dev/null
@@ -1,37 +0,0 @@
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# base things off the testing image without dependencies warmed up
-FROM spod/cassandra-testing-ubuntu1810-java11
-MAINTAINER Stefan Podkowinski <[email protected]>
-
-USER cassandra
-ENV HOME /home/cassandra
-WORKDIR /home/cassandra
-
-# Fetch the maven dependencies in advance since this tends to fail at runtime
-ARG CASSANDRA_GIT_URL=https://gitbox.apache.org/repos/asf/cassandra.git
-RUN /bin/bash -c "git clone ${CASSANDRA_GIT_URL} ~/cassandra && \
- cd ~/cassandra && ant maven-ant-tasks-retrieve-build && \
- git checkout origin/cassandra-3.11 && ant maven-ant-tasks-retrieve-build
&& \
- git checkout origin/cassandra-3.0 && ant maven-ant-tasks-retrieve-build &&
\
- git checkout origin/cassandra-2.2 && ant maven-ant-tasks-retrieve-build &&
\
- git checkout origin/cassandra-2.1 && ant maven-ant-tasks-retrieve-build &&
\
- rm -fr ~/cassandra"
-
-# Initialize the CCM git repo as well as this also can fail to clone
-RUN /bin/bash -c "source ~/env/bin/activate && \
- ccm create -n 1 -v git:trunk test && ccm remove test && \
- ccm create -n 1 -v git:cassandra-3.11 test && ccm remove test && \
- ccm create -n 1 -v git:cassandra-3.0 test && ccm remove test && \
- ccm create -n 1 -v git:cassandra-2.2 test && ccm remove test && \
- ccm create -n 1 -v git:cassandra-2.1 test && ccm remove test"
diff --git a/docker/testing/ubuntu18_j11.docker
b/docker/testing/ubuntu18_j11.docker
deleted file mode 100644
index 88a80d6..0000000
--- a/docker/testing/ubuntu18_j11.docker
+++ /dev/null
@@ -1,112 +0,0 @@
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# base things off the latest LTS Ubuntu Release (18.04)
-FROM ubuntu:18.04
-MAINTAINER Stefan Podkowinski <[email protected]>
-
-# install our python dependencies and some other stuff we need
-# libev4 libev-dev are for the python driver
-
-RUN export DEBIAN_FRONTEND=noninteractive && \
- apt-get update && \
- apt-get install -y --no-install-recommends software-properties-common
apt-utils vim
-
-RUN export DEBIAN_FRONTEND=noninteractive && \
- apt-get update && \
- apt-get install -y git-core python2.7 python3.6 python3.6-venv
python3.6-dev python3-pip net-tools libev4 libev-dev wget gcc
-
-# solves warning: "jemalloc shared library could not be preloaded to speed up
memory allocations"
-RUN export DEBIAN_FRONTEND=noninteractive && \
- apt-get update && \
- apt-get install -y --no-install-recommends libjemalloc1
-
-# install dumb-init as minimal init system
-RUN export DEBIAN_FRONTEND=noninteractive && \
- apt-get update && \
- apt-get install -y dumb-init
-
-# generate locales for the standard en_US.UTF8 value we use for testing
-RUN export DEBIAN_FRONTEND=noninteractive && \
- apt-get update && \
- apt-get install -y locales && \
- locale-gen en_US.UTF-8
-
-# as we only need the requirements.txt file from the dtest repo, let's just
get it from GitHub as a raw asset
-# so we can avoid needing to clone the entire repo just to get this file
-ADD
https://raw.githubusercontent.com/apache/cassandra-dtest/trunk/requirements.txt
/opt
-RUN chmod 0644 /opt/requirements.txt
-
-# now setup python via virtualenv with all of the python dependencies we need
according to requirements.txt
-RUN pip3 install virtualenv
-RUN pip3 install --upgrade wheel
-
-# openjdk + ant
-RUN export DEBIAN_FRONTEND=noninteractive && \
- apt-get update && \
- apt-get install -y --no-install-recommends openjdk-8-jdk openjdk-11-jdk
ant ant-optional
-
-# make Java 8 the default executable (we use to run all tests against Java 8)
-RUN update-java-alternatives -s java-1.8.0-openjdk-amd64
-
-# setup our user -- if we don't do this docker will default to root and
Cassandra will fail to start
-# as we appear to have a check that the user isn't starting Cassandra as root
-RUN export DEBIAN_FRONTEND=noninteractive && \
- apt-get install sudo && \
- adduser --disabled-password --gecos "" cassandra && \
- echo "cassandra ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/cassandra && \
- chmod 0440 /etc/sudoers.d/cassandra
-
-# fix up permissions on the cassandra home dir
-RUN chown -R cassandra:cassandra /home/cassandra
-
-# switch to the cassandra user... we are all done running things as root
-USER cassandra
-ENV HOME /home/cassandra
-WORKDIR /home/cassandra
-
-# Add environment variables for Ant and Java and add them to the PATH
-RUN echo 'export ANT_HOME=/usr/share/ant' >> /home/cassandra/.bashrc && \
- echo 'export JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64' >>
/home/cassandra/.bashrc && \
- echo 'export JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64' >>
/home/cassandra/.bashrc && \
- echo 'export JAVA_HOME=$JAVA8_HOME' >> /home/cassandra/.bashrc
-
-ENV ANT_HOME=/usr/share/ant \
- JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
- JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
- JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64
-
-# run pip commands and setup virtualenv (note we do this after we switch to
cassandra user so we
-# setup the virtualenv for the cassandra user and not the root user by
accident)
-RUN virtualenv --python=python3.6 --no-site-packages env
-RUN chmod +x env/bin/activate
-RUN /bin/bash -c "source ~/env/bin/activate && pip3 install Cython && pip3
install -r /opt/requirements.txt && pip3 freeze --user"
-
-# we need to make SSH less strict to prevent various dtests from failing when
they attempt to
-# git clone a given commit/tag/etc
-# upgrading node1 to github:apache/18cdd391ec27d16daf775f928902f5a421c415e3
-# [email protected]:apache/cassandra.git
github:apache/18cdd391ec27d16daf775f928902f5a421c415e3
-# 23:47:08,993 ccm INFO Cloning Cassandra...
-# The authenticity of host 'github.com (192.30.253.112)' can't be established.
-# RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
-# Are you sure you want to continue connecting (yes/no)?
-RUN mkdir ~/.ssh
-RUN echo 'Host *\n UserKnownHostsFile /dev/null\n StrictHostKeyChecking no' >
~/.ssh/config
-RUN chown cassandra:cassandra ~/.ssh
-RUN chown cassandra:cassandra ~/.ssh/config
-RUN chmod 600 ~/.ssh/config
-
-# mark "/tmp" as a volume so it will get mounted as an ext4 mount and not
-# the stupid aufs/CoW stuff that the actual docker container mounts will have.
-# we've been seeing 3+ minute hangs when calling sync on an aufs backed mount
-# so it greatly makes tests flaky as things can hang basically anywhere
-VOLUME ["/tmp"]
diff --git a/docker/testing/ubuntu1910_j11.docker
b/docker/testing/ubuntu1910_j11.docker
index 5351b1e..7c28c14 100644
--- a/docker/testing/ubuntu1910_j11.docker
+++ b/docker/testing/ubuntu1910_j11.docker
@@ -11,12 +11,13 @@
# limitations under the License.
FROM ubuntu:19.10
-MAINTAINER Sam Tunnicliffe <[email protected]>
+MAINTAINER Apache Cassandra <[email protected]>
# install our python dependencies and some other stuff we need
# libev4 libev-dev are for the python driver / libssl-dev is for python3.6
RUN export DEBIAN_FRONTEND=noninteractive && \
+ sed -i -re
's/([a-z]{2}\.)?archive.ubuntu.com|security.ubuntu.com/old-releases.ubuntu.com/g'
/etc/apt/sources.list && \
apt-get update && \
apt-get install -y --no-install-recommends software-properties-common
apt-utils vim
diff --git a/docker/testing/ubuntu1910_j11_w_dependencies.docker
b/docker/testing/ubuntu1910_j11_w_dependencies.docker
index 4ea6fb5..2a4fd16 100644
--- a/docker/testing/ubuntu1910_j11_w_dependencies.docker
+++ b/docker/testing/ubuntu1910_j11_w_dependencies.docker
@@ -11,8 +11,8 @@
# limitations under the License.
# base things off the testing image without dependencies warmed up
-FROM beobal/cassandra-testing-ubuntu1910-java11
-MAINTAINER Sam Tunnicliffe <[email protected]>
+FROM apache/cassandra-testing-ubuntu1910-java11
+MAINTAINER Apache Cassandra <[email protected]>
USER cassandra
ENV HOME /home/cassandra
diff --git a/docker/testing/ubuntu1810_j11.docker
b/docker/testing/ubuntu2004_j11.docker
similarity index 75%
rename from docker/testing/ubuntu1810_j11.docker
rename to docker/testing/ubuntu2004_j11.docker
index 07d68ba..fc4a6ed 100644
--- a/docker/testing/ubuntu1810_j11.docker
+++ b/docker/testing/ubuntu2004_j11.docker
@@ -10,20 +10,31 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# base things off the latest LTS Ubuntu Release (18.04)
-FROM ubuntu:18.10
-MAINTAINER Stefan Podkowinski <[email protected]>
+FROM ubuntu:20.04
+MAINTAINER Apache Cassandra <[email protected]>
# install our python dependencies and some other stuff we need
-# libev4 libev-dev are for the python driver
+# libev4 libev-dev are for the python driver / libssl-dev is for python3.6
RUN export DEBIAN_FRONTEND=noninteractive && \
apt-get update && \
apt-get install -y --no-install-recommends software-properties-common
apt-utils vim
RUN export DEBIAN_FRONTEND=noninteractive && \
+ add-apt-repository -y ppa:deadsnakes/ppa && \
apt-get update && \
- apt-get install -y git-core python2.7 python3.6 python3.6-venv
python3.6-dev python3-pip net-tools libev4 libev-dev wget gcc
+ apt-get install -y git-core python2.7 python3-pip \
+ python3.6 python3.6-venv python3.6-dev \
+ python3.7 python3.7-venv python3.7-dev \
+ python3.8 python3.8-venv python3.8-dev \
+ net-tools libev4 libev-dev wget gcc libssl-dev
+
+RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2
+RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.7 3
+RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.8 4
+RUN python3.6 -m pip install --upgrade pip
+RUN python3.7 -m pip install --upgrade pip
+RUN python3.8 -m pip install --upgrade pip
# solves warning: "jemalloc shared library could not be preloaded to speed up
memory allocations"
RUN export DEBIAN_FRONTEND=noninteractive && \
@@ -86,10 +97,18 @@ ENV ANT_HOME=/usr/share/ant \
JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64
# run pip commands and setup virtualenv (note we do this after we switch to
cassandra user so we
-# setup the virtualenv for the cassandra user and not the root user by
accident)
-RUN virtualenv --python=python3.6 --no-site-packages env
-RUN chmod +x env/bin/activate
-RUN /bin/bash -c "source ~/env/bin/activate && pip3 install Cython && pip3
install -r /opt/requirements.txt && pip3 freeze --user"
+# setup the virtualenv for the cassandra user and not the root user by
accident) for Python 3.6/3.7/3.8
+RUN virtualenv --python=python3.6 env3.6
+RUN chmod +x env3.6/bin/activate
+RUN /bin/bash -c "source ~/env3.6/bin/activate && pip3 install --upgrade pip
&& pip3 install Cython && pip3 install -r /opt/requirements.txt && pip3 freeze
--user"
+
+RUN virtualenv --python=python3.7 env3.7
+RUN chmod +x env3.7/bin/activate
+RUN /bin/bash -c "source ~/env3.7/bin/activate && pip3 install --upgrade pip
&& pip3 install Cython && pip3 install -r /opt/requirements.txt && pip3 freeze
--user"
+
+RUN virtualenv --python=python3.8 env3.8
+RUN chmod +x env3.8/bin/activate
+RUN /bin/bash -c "source ~/env3.8/bin/activate && pip3 install --upgrade pip
&& pip3 install Cython && pip3 install -r /opt/requirements.txt && pip3 freeze
--user"
# we need to make SSH less strict to prevent various dtests from failing when
they attempt to
# git clone a given commit/tag/etc
diff --git a/docker/testing/ubuntu18_j11_w_dependencies.docker
b/docker/testing/ubuntu2004_j11_w_dependencies.docker
similarity index 91%
rename from docker/testing/ubuntu18_j11_w_dependencies.docker
rename to docker/testing/ubuntu2004_j11_w_dependencies.docker
index 0aaead9..c9ce72f 100644
--- a/docker/testing/ubuntu18_j11_w_dependencies.docker
+++ b/docker/testing/ubuntu2004_j11_w_dependencies.docker
@@ -11,8 +11,8 @@
# limitations under the License.
# base things off the testing image without dependencies warmed up
-FROM aweisberg/cassandra-testing-ubuntu18-java11
-MAINTAINER Stefan Podkowinski <[email protected]>
+FROM apache/cassandra-testing-ubuntu2004-java11
+MAINTAINER Apache Cassandra <[email protected]>
USER cassandra
ENV HOME /home/cassandra
@@ -29,7 +29,7 @@ RUN /bin/bash -c "git clone ${CASSANDRA_GIT_URL} ~/cassandra
&& \
rm -fr ~/cassandra"
# Initialize the CCM git repo as well as this also can fail to clone
-RUN /bin/bash -c "source ~/env/bin/activate && \
+RUN /bin/bash -c "source ~/env3.6/bin/activate && \
ccm create -n 1 -v git:trunk test && ccm remove test && \
ccm create -n 1 -v git:cassandra-3.11 test && ccm remove test && \
ccm create -n 1 -v git:cassandra-3.0 test && ccm remove test && \
diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy
b/jenkins-dsl/cassandra_job_dsl_seed.groovy
index 32a4bbf..2b5eca6 100644
--- a/jenkins-dsl/cassandra_job_dsl_seed.groovy
+++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy
@@ -59,7 +59,7 @@ def dtestTargets = ['dtest', 'dtest-novnode',
'dtest-offheap', 'dtest-large', 'd
if(binding.hasVariable("CASSANDRA_DTEST_TEST_TARGETS")) {
dtestTargets = "${CASSANDRA_DTEST_TEST_TARGETS}".split(",")
}
-def dtestDockerImage = 'beobal/cassandra-testing-ubuntu1910-java11'
+def dtestDockerImage = 'apache/cassandra-testing-ubuntu2004-java11'
if(binding.hasVariable("CASSANDRA_DOCKER_IMAGE")) {
dtestDockerImage = "${CASSANDRA_DOCKER_IMAGE}"
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
