Re: [yocto] [layerindex-web][PATCH 1/6] Add docker-compose file to create full layerindex stack of MariaDB, RabbitMQ and Nginx

2018-07-10 Thread Paul Eggleton
Hi Konrad

On Tuesday, 26 June 2018 7:41:30 PM CEST Konrad Scherer wrote:
> Lots of new features added:
> 
> - Layerindex runs as unprivileged user inside container
> 
> - Celery worker is started before gunicorn
> 
> - Entrypoint script supports changing RabbitMQ location
> 
> - Entrypoint script support initialization of database and superuser
> 
> - Reverse Proxy uses https with self signed certs by default and
>   supports Let's Encrypt certs (not enabled by default)
> 
> - Move docker image to debian stretch and python3
> 
> - Remove build tools after installation to reduce the image
>   to under 500MB in size

This is quite nice, thanks!

Coincidentally, both myself and Michael Halstead have had a go at cleaning up
and improving the Docker setup, so I have to do a little reconciliation
between your and his changes - see here for his:

https://github.com/halstead/layerindex-web/commit/b9791710ff97550fa9110ab89a70c42b1fc86581

I think we probably want to break all of this down into a set of discrete 
commits rather than one big one, and then it'll be a bit clearer. I'll do a
first pass and come back to you both.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre


-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto


[yocto] [layerindex-web][PATCH 1/6] Add docker-compose file to create full layerindex stack of MariaDB, RabbitMQ and Nginx

2018-06-26 Thread Konrad Scherer
Lots of new features added:

- Layerindex runs as unprivileged user inside container

- Celery worker is started before gunicorn

- Entrypoint script supports changing RabbitMQ location

- Entrypoint script support initialization of database and superuser

- Reverse Proxy uses https with self signed certs by default and
  supports Let's Encrypt certs (not enabled by default)

- Move docker image to debian stretch and python3

- Remove build tools after installation to reduce the image
  to under 500MB in size

Signed-off-by: Konrad Scherer 
---
 Dockerfile |  78 
 docker/README  |  56 +
 docker/docker-compose.yaml | 111 +
 docker/entrypoint.sh   |  32 +
 docker/mariadb_settings.py | 246 +
 5 files changed, 470 insertions(+), 53 deletions(-)
 create mode 100644 docker/docker-compose.yaml
 create mode 100755 docker/entrypoint.sh
 create mode 100644 docker/mariadb_settings.py

diff --git a/Dockerfile b/Dockerfile
index 9bb251e..6f5ad16 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,40 +1,64 @@
-FROM buildpack-deps:latest
+FROM debian:stretch
 MAINTAINER Michael Halstead 
 
 EXPOSE 80
-ENV PYTHONUNBUFFERED 1
+ENV PYTHONUNBUFFERED=1 \
+LANG=en_US.UTF-8 \
+LC_ALL=en_US.UTF-8 \
+LC_CTYPE=en_US.UTF-8
+
 ## Uncomment to set proxy ENVVARS within container
 #ENV http_proxy http://your.proxy.server:port
 #ENV https_proxy https://your.proxy.server:port
 
-RUN apt-get update
-RUN apt-get install -y --no-install-recommends \
-   python-pip \
-   python-mysqldb \
-   python-dev \
-   python-imaging \
-   rabbitmq-server \
-   netcat-openbsd \
-   vim \
-   && rm -rf /var/lib/apt/lists/*
-RUN pip install --upgrade pip
-RUN pip install gunicorn
-RUN pip install setuptools
-CMD mkdir /opt/workdir
+ADD requirements.txt /
+
+RUN apt-get update \
+&& apt-get install -y --no-install-recommends \
+  autoconf \
+  g++ \
+  gcc \
+  make \
+  python3-pip \
+  python3-dev \
+  python3-pil \
+  python3-mysqldb \
+  python3-setuptools \
+  netcat-openbsd \
+  libjpeg-dev \
+  vim git curl locales libmariadbclient-dev \
+&& echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
+&& locale-gen en_US.UTF-8 \
+&& update-locale \
+&& mkdir /opt/workdir \
+&& pip3 install wheel gunicorn \
+&& pip3 install -r /requirements.txt \
+&& apt-get purge -y g++ make python3-dev autoconf libmariadbclient-dev \
+&& apt-get autoremove -y \
+&& rm -rf /var/lib/apt/lists/* \
+&& apt-get clean \
+&& groupadd user \
+&& useradd --create-home --home-dir /home/user -g user user
+
 ADD . /opt/layerindex
-RUN pip install -r /opt/layerindex/requirements.txt
-ADD settings.py /opt/layerindex/settings.py
+
+# Copy static resouces to static dir so they can be served by nginx
+RUN rm -f /opt/layerindex/layerindex/static/admin \
+&& cp -r 
/usr/local/lib/python3.5/dist-packages/django/contrib/admin/static/admin/ \
+/opt/layerindex/layerindex/static/ \
+&& rm -f /opt/layerindex/layerindex/static/rest_framework \
+&& cp -r 
/usr/local/lib/python3.5/dist-packages/rest_framework/static/rest_framework/ \
+/opt/layerindex/layerindex/static/ \
+&& chown -R user:user /opt/layerindex \
+&& mkdir /opt/layers && chown -R user:user /opt/layers
+
 ADD docker/updatelayers.sh /opt/updatelayers.sh
 ADD docker/migrate.sh /opt/migrate.sh
 
-## Uncomment to add a .gitconfig file within container
-#ADD docker/.gitconfig /root/.gitconfig
-## Uncomment to add a proxy script within container, if you choose to
-## do so, you will also have to edit .gitconfig appropriately
-#ADD docker/git-proxy /opt/bin/git-proxy
+# Add entrypoint to start celery worker and gnuicorn
+ADD docker/entrypoint.sh /entrypoint.sh
 
-# Start Gunicorn
-CMD ["/usr/local/bin/gunicorn", "wsgi:application", "--workers=4", 
"--bind=:5000", "--log-level=debug", "--chdir=/opt/layerindex"]
+# Run gunicorn and celery as unprivileged user
+USER user
 
-# Start Celery
-CMD ["/usr/local/bin/celery", "-A", "layerindex.tasks", "worker", 
"--loglevel=info", "--workdir=/opt/layerindex"]
+ENTRYPOINT ["/entrypoint.sh"]
diff --git a/docker/README b/docker/README
index 14bc392..dc5c37c 100644
--- a/docker/README
+++ b/docker/README
@@ -1,26 +1,30 @@
-## This is set up to make a cluster of three containers. First we build two 
from the root of the repo.
-docker build -t halstead/layerindex-app .
-docker build -t halstead/layerindex-web -f Dockerfile.web .
-
-## Start a database server. We use MariaDB in production.
-## In order to configure your settings.py file to use this database server, 
use:
-##   'ENGINE': 'django.db.backends.mysql',
-##   'NAME': 'layersdb',
-##   'USER': 'root',
-##   'PASSWORD': 'testingpw',
-##   'HOST': 'layersdb',
-##   'PORT': '',
-docker run -d --name layerdb -e MYSQL_ROOT_PASSWORD=testingpw -e 
MYSQL_DATABASE=layersdb mariadb
-