On 09 Aug 14:27, Daniel Axtens wrote: > This makes it possible to use Docker and docker-compose for development > as an alternative to Vagrant. > > I quite liked vagrant a couple of years ago, but currently: > > * Trying to install VirtualBox on Ubuntu wants me to disable > Secure Boot, and I don't want to do that. > > * Trying to use the libvirt plugin for vagrant requires I pick > from a very small set of possible images, and requires that I > install the upstream vagrant rather than the vagrant shipped > with Ubuntu 16.04 > > * I find docker containers faster to work with and more transparent. > > So I've done the work to make docker work for Patchwork development. > This doesn't break or in any way interfere with using Vagrant, it just > provides an alternative. > > It includes support for headless selenium tests using Chromium.
Excellent work. I've a couple of minor requests below, then we're good to go I think. > Signed-off-by: Daniel Axtens <[email protected]> > > --- > > v2: chrome -> chromium > v3: The following, thanks largely to Andrew: > - various minor tweaks > - wait longer for db to come up > - grant patchwork user permissions to test db > --- > .dockerignore | 3 ++ > README.md | 38 +++++++++++++++++++++- > docker-compose.yml | 23 +++++++++++++ > docker/Dockerfile | 46 ++++++++++++++++++++++++++ > docker/bashrc | 5 +++ > docker/db/.dockerignore | 1 + > docker/db/.gitignore | 1 + > docker/db/Dockerfile | 10 ++++++ > docker/entrypoint.sh | 86 > +++++++++++++++++++++++++++++++++++++++++++++++++ > 9 files changed, 212 insertions(+), 1 deletion(-) > create mode 100644 .dockerignore > create mode 100644 docker-compose.yml > create mode 100644 docker/Dockerfile > create mode 100644 docker/bashrc > create mode 100644 docker/db/.dockerignore > create mode 100644 docker/db/.gitignore > create mode 100644 docker/db/Dockerfile > create mode 100755 docker/entrypoint.sh > > diff --git a/.dockerignore b/.dockerignore > new file mode 100644 > index 000000000000..24473bbe1237 > --- /dev/null > +++ b/.dockerignore > @@ -0,0 +1,3 @@ > +.vagrant > +docker/db > + > diff --git a/README.md b/README.md > index c83e7d31f1db..3c25f04d7c8d 100644 > --- a/README.md > +++ b/README.md > @@ -20,7 +20,7 @@ subsystems of the Linux kernel. Although Patchwork has been > developed with the > kernel workflow in mind, the aim is to be flexible enough to suit the > majority > of community projects. > > -# Development Installation > +# Development Installation using Vagrant > > 1. Install [**Vagrant**][ref-vagrant] > 2. Clone this repo: > @@ -32,6 +32,42 @@ of community projects. > $ cd patchwork > $ vagrant up > > +# Development Installation using Docker > + > +1. Install Docker and docker-compose. > +2. Clone this repo, as with vagrant. > +3. Build the images. This will download over 200MB from the internet: > + > + $ docker-compose build > + > +4. Run as follows: > + > + * Regular server: > + > + $ docker-compose up Can you include only up to here in the README and move the rest to 'docs/development.md'? I'd like to keep the README short and sweet. A reference to the docs (perhaps the readthedocs hosted version) might be helpful. http://patchwork.readthedocs.io/en/latest/development/ > + > + This will be visible on http://localhost:8000/. > + > + * Shell: > + > + $ docker-compose run --rm web --shell > + > + * Quick test (not including selenium UI interaction tests): > + > + $ docker-compose run --rm web --quick-test > + > + * Full tests, including selenium, run headlessly: > + > + $ docker-compose run --rm web --test > + > + * To reset the database before beginning, add `--reset` to the command > line after `web` and before any other arguments. > + > + * If you want to run non-headless tests, you'll need something like this > ugly hack: > + > + $ docker run -it --rm -v (pwd):/home/patchwork/patchwork/ --link > patchwork_db_1:db -p 8000:8000 -v /tmp/.X11-unix:/tmp/.X11-unix -e > PW_TEST_DB_HOST=db -e DISPLAY patchwork_web bash > + > +With both vagrant and docker, any edits to the project files made locally > are immediately visible to the VM/container, and so should be picked up by > the Django auto-reloader. Wrap text and commands (with slashes) at 80 characters. > + > # Talks and Presentations > > * [**A New Patchwork**][pdf-fosdem] - FOSDEM 2016 > diff --git a/docker-compose.yml b/docker-compose.yml > new file mode 100644 > index 000000000000..881a72ad6de1 > --- /dev/null > +++ b/docker-compose.yml > @@ -0,0 +1,23 @@ > +# the version of docker-compose shipped in ubuntu 16.04 is > +# 1.5.2, which doesn't support version 2 syntax. Yay! > +# also, v1 doesn't support explicit build args, so if you're not > +# uid 1000, you will either need to manually hack the Dockerfile > +# or upgrade to v2 and use the build-arg to override it. > + > +db: > + build: docker/db > + volumes: > + - ./docker/db/data:/var/lib/mysql > +web: > + build: . > + dockerfile: ./docker/Dockerfile > + command: python3 manage.py runserver 0.0.0.0:8000 > + volumes: > + - .:/home/patchwork/patchwork/ > + ports: > + - "8000:8000" > + links: > + - db > + environment: > + - PW_TEST_DB_HOST=db > + - PW_TEST_DB_PORT=3306 > diff --git a/docker/Dockerfile b/docker/Dockerfile > new file mode 100644 > index 000000000000..ea2a8fc5ab77 > --- /dev/null > +++ b/docker/Dockerfile Could you move the 'docker' folder to 'tools/docker', please? Perhaps rename 'tools/install' to 'tools/vagrant' while at it? I think this folder is a natural fit for these scripts. > @@ -0,0 +1,46 @@ > +FROM ubuntu > + > +ARG UID=1000 > + > +ENV PROJECT_HOME /home/patchwork/patchwork > + > +ENV db_user root > +ENV db_pass password > + > +ENV DJANGO_SETTINGS_MODULE patchwork.settings.dev > +ENV DEBIAN_FRONTEND noninteractive > +ENV PYTHONUNBUFFERED 1 > + > +# System > +RUN apt-get update -qq && \ > + apt-get install -y --no-install-recommends \ > + python-dev python-pip python-setuptools python-wheel \ > + python3-dev python3-pip python3-setuptools python3-wheel \ > + libmysqlclient-dev mysql-client curl unzip xvfb chromium-chromedriver \ > + chromium-browser build-essential && \ > + ln -s /usr/lib/chromium-browser/chromedriver /usr/bin/ > + > +# User > +RUN useradd --uid=$UID --create-home patchwork > + > +# Python requirements. > +# If you update requirements, you should rebuild the container. > +# entrypoint.sh will prompt you to do this. > +COPY requirements-*.txt /tmp/ > +RUN pip3 install virtualenv tox && \ > + pip3 install -r /tmp/requirements-dev.txt > +# we deliberately leave the requirements files in tmp so we can > +# ping the user in entrypoint.sh if the change them! > + > +COPY docker/bashrc /tmp/bashrc > + > +# we put the code in ~/patchwork rather than ~ so that we > +# can put in these bashrc snippets without dirtying the > +# working directory > +RUN cat /tmp/bashrc >> /home/patchwork/.bashrc > + > +COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh > + > +ENTRYPOINT ["/usr/local/bin/entrypoint.sh"] > +USER patchwork > +WORKDIR /home/patchwork/patchwork > diff --git a/docker/bashrc b/docker/bashrc > new file mode 100644 > index 000000000000..eb2ed7dddb85 > --- /dev/null > +++ b/docker/bashrc > @@ -0,0 +1,5 @@ > +# This snippet is appended to ~/.bashrc when the container is created > + > +alias runserver='python3 $PROJECT_HOME/manage.py runserver 0.0.0.0:8000' > +alias createsu='python3 $PROJECT_HOME/manage.py createsuperuser' > + > diff --git a/docker/db/.dockerignore b/docker/db/.dockerignore > new file mode 100644 > index 000000000000..1269488f7fb1 > --- /dev/null > +++ b/docker/db/.dockerignore > @@ -0,0 +1 @@ > +data > diff --git a/docker/db/.gitignore b/docker/db/.gitignore > new file mode 100644 > index 000000000000..60baa9cb833f > --- /dev/null > +++ b/docker/db/.gitignore > @@ -0,0 +1 @@ > +data/* > diff --git a/docker/db/Dockerfile b/docker/db/Dockerfile > new file mode 100644 > index 000000000000..5df9b5acb486 > --- /dev/null > +++ b/docker/db/Dockerfile > @@ -0,0 +1,10 @@ > +FROM mysql:5.7 > + > +ENV MYSQL_ROOT_PASSWORD password > +ENV MYSQL_USER patchwork > +ENV MYSQL_PASSWORD password > + > +# We don't want to use the MYSQL_DATABASE env here because > +# we want to be able to create the database with UTF-8 explictly. > +# We also can't load in the data because it's in XML, yay. > + > diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh > new file mode 100755 > index 000000000000..0966cf2ec432 > --- /dev/null > +++ b/docker/entrypoint.sh > @@ -0,0 +1,86 @@ > +#!/bin/bash > +set -euo pipefail > + > +# functions > +test_db_connection() { > + mysqladmin -h $PW_TEST_DB_HOST -u patchwork --password=password ping > > /dev/null 2> /dev/null > +} > + > +reset_data() { > + mysql -u$db_user -p$db_pass -h $PW_TEST_DB_HOST << EOF > +DROP DATABASE IF EXISTS patchwork; > +CREATE DATABASE patchwork CHARACTER SET utf8; > +GRANT ALL ON patchwork.* TO 'patchwork' IDENTIFIED BY 'password'; > +GRANT ALL PRIVILEGES ON test_patchwork.* TO 'patchwork'@'%'; > +FLUSH PRIVILEGES; > +EOF > + > + # load initial data > + python3 $PROJECT_HOME/manage.py migrate #> /dev/null > + python3 $PROJECT_HOME/manage.py loaddata default_tags #> /dev/null > + python3 $PROJECT_HOME/manage.py loaddata default_states #> /dev/null > + python3 $PROJECT_HOME/manage.py loaddata default_projects #> /dev/null > +} > + > + > +# The script begins! > + > +# check if patchwork is mounted. Checking if we exist is a > +# very good start! > +if [ ! -f ~patchwork/patchwork/docker/entrypoint.sh ]; then > + echo "The patchwork directory doesn't seem to be mounted!" > + echo "Are you using docker-compose?" > + echo "If not, you need -v PATH_TO_PATCHWORK:/home/patchwork/patchwork" > + exit 1 > +fi > + > +# check if we need to rebuild because requirements changed > +for x in /tmp/requirements-*.txt; do > + if ! cmp $x ~/patchwork/$(basename $x); then > + echo "A requirements file has changed." > + echo "Please rebuild the patchwork image:" > + echo " docker-compose build web" > + exit 1 Spaces, not tabs, please: I'm no Richard Hendriks [1] :) [1] https://www.youtube.com/watch?v=SsoOG6ZeyUI > + fi > +done > + > +# check if mysql is connected > +if ! test_db_connection; then > + echo "MySQL seems not to be connected, or the patchwork user is broken" > + echo "MySQL may still be starting. Waiting 5 seconds." > + sleep 5 > + if ! test_db_connection; then > + echo "Still cannot connect to MySQL." > + echo "Maybe you are starting the db for the first time. Waiting 15 > seconds." > + sleep 15 > + if ! test_db_connection; then > + echo "Still cannot connect to MySQL. Giving up." > + echo "Are you using docker-compose? If not, have you set up the > link correctly?" > + exit 1 > + fi > + fi > +fi > + > +# rebuild mysql db > +# do this on --reset or if the db doesn't exist > +if [[ "$1" == "--reset" ]]; then > + shift > + reset_data > +elif ! ( echo ';' | mysql -h db -u patchwork -ppassword patchwork 2> > /dev/null ); then > + reset_data > +fi > + > +if [ $# -eq 0 ]; then > + # we probably ran with --reset and nothing else > + # just exit cleanly > + exit 0 > +elif [ "$1" == "--shell" ]; then > + exec bash > +elif [ "$1" == "--quick-test" ]; then > + export PW_SKIP_BROWSER_TESTS=yes > + python3 manage.py test > +elif [ "$1" == "--test" ]; then > + xvfb-run --server-args='-screen 0, 1024x768x16' python3 manage.py test > +else # run whatever CMD is set to > + $@ > +fi > -- > 2.7.4 > _______________________________________________ Patchwork mailing list [email protected] https://lists.ozlabs.org/listinfo/patchwork
