Hi all, I've been discussing this with Dave for about a month now. Today I finally present a proposed patch to update pgadmin4's Docker packaging.
Key features of this update: - Main image is based on python:3.6-alpine3.7. Using Alpine linux leads to much smaller image - All build is done with Docker multi-stage build. First of all build the frontend in node:6 image, then build Sphinx documentation in separate Python container and in the end just install all dependencies in a clean python:3.6-alpine3.7 image, so that it does not have any leftovers from the build process and we don't rely on any tools available on the host. - Use Gunicorn (http://gunicorn.org) as lightweight HTTP / WSGI server. Gunicorn supports both HTTP and HTTPS. - Install Alpine postgresql-client package, which includes pg_dump and other tools and config PgAdmin to find these tools by default - Byte-compile all PgAdmin Python code in Dockerfile with optimization (-O) enabled. This way Python does not have to compile modules on each container restart and consume space in overlay fs Please find attached patch from "git format-patch".
From 67e387525a7c832958858c7ec1a1b7076382090e Mon Sep 17 00:00:00 2001 From: Maxim Koltsov <kolma...@gmail.com> Date: Sat, 31 Mar 2018 20:37:51 +0300 Subject: [PATCH] Re-make Docker container packaging Key features of this update: - Main image is based on python:3.6-alpine3.7. Using Alpine linux leads to much smaller image - All build is done with Docker multi-stage build. First of all build the frontend in node:6 image, then build Sphinx documentation in separate Python container and in the end just install all dependencies in a clean python:3.6-alpine3.7 image, so that it does not have any leftovers from the build process and we don't rely on any tools available on the host. - Use Gunicorn (http://gunicorn.org) as lightweight HTTP / WSGI server. Gunicorn supports both HTTP and HTTPS. - Install Alpine postgresql-client package, which includes pg_dump and other tools and config PgAdmin to find these tools by default - Byte-compile all PgAdmin Python code in Dockerfile with optimization (-O) enabled. This way Python does not have to compile modules on each container restart and consume space in overlay fs --- pkg/docker/Dockerfile | 89 ++++++++++++++++++++++++--------------------- pkg/docker/README | 58 ++++++++++++++--------------- pkg/docker/build.sh | 57 +++++------------------------ pkg/docker/config_distro.py | 4 ++ pkg/docker/entry.sh | 29 --------------- pkg/docker/entrypoint.sh | 21 +++++++++++ pkg/docker/pgadmin4.conf.j2 | 43 ---------------------- pkg/docker/run_pgadmin.py | 4 ++ 8 files changed, 112 insertions(+), 193 deletions(-) create mode 100644 pkg/docker/config_distro.py delete mode 100644 pkg/docker/entry.sh create mode 100755 pkg/docker/entrypoint.sh delete mode 100644 pkg/docker/pgadmin4.conf.j2 create mode 100644 pkg/docker/run_pgadmin.py diff --git a/pkg/docker/Dockerfile b/pkg/docker/Dockerfile index 1c1dde27..083dbd60 100644 --- a/pkg/docker/Dockerfile +++ b/pkg/docker/Dockerfile @@ -7,58 +7,63 @@ # ######################################################################### -# Get the basics out of the way -FROM centos:latest +# First of all, build frontend with NodeJS in a separate builder container +# Node-6 with ABI v48 is supported by all needed C++ packages +FROM node:6 AS node-builder -LABEL name="pgAdmin 4" \ - vendor="The pgAdmin Development Team" \ - license="PostgreSQL" +COPY ./pgadmin4/web/ /pgadmin4/web/ +WORKDIR /pgadmin4/web -# We only need the web/ directory, and a few other things -COPY web /var/www/pgadmin -COPY requirements.txt /var/www/pgadmin +RUN yarn install --cache-folder ./ycache --verbose && \ + yarn run bundle && \ + rm -rf ./ycache ./pgadmin/static/js/generated/.cache -# Install everything we need. Use easy_install to get pip, to avoid setting up EPEL -RUN yum install -y python-setuptools python-devel httpd mod_wsgi mod_ssl gcc -RUN easy_install pip -RUN pip install j2cli +# Build Sphinx documentation in separate container +FROM python:3.6-alpine3.7 as docs-builder -# Now install the Python runtime dependencies -RUN pip install -r /var/www/pgadmin/requirements.txt +# Install only dependencies absolutely required for documentation building +RUN apk add --no-cache make +RUN pip install --no-cache-dir \ + sphinx flask_babel flask_security flask_paranoid python-dateutil flask_sqlalchemy \ + flask_gravatar simplejson -# Create required directories for config +COPY ./pgadmin4/ /pgadmin4 +RUN LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 make -C /pgadmin4/docs/en_US -f Makefile.sphinx html -# Create required directories for running -RUN mkdir -p /var/log/pgadmin -RUN chown -R apache /var/log/pgadmin -RUN mkdir -p /var/lib/pgadmin -RUN chown -R apache /var/lib/pgadmin -RUN mkdir -p /certs -RUN chown -R apache /certs -RUN chmod 700 /certs +# Then install backend, copy static files and set up entrypoint +# Need alpine3.7 to get pg_dump and friends in postgresql-client package +FROM python:3.6-alpine3.7 -# Push logs to the container's output streams -RUN ln -sf /proc/self/fd/1 /var/log/httpd/access_log && \ - ln -sf /proc/self/fd/1 /var/log/httpd/ssl_access_log && \ - ln -sf /proc/self/fd/2 /var/log/httpd/error_log && \ - ln -sf /proc/self/fd/2 /var/log/httpd/ssl_error_log +RUN pip --no-cache-dir install gunicorn +RUN apk add --no-cache postgresql-client postgresql-libs -# Apache config time -RUN mkdir -p /templates -COPY pgadmin4.conf.j2 /templates/ -COPY entry.sh / +# Install build-dependencies, build & install C extensions and purge deps in one RUN step +# so that deps do not increase the size of resulting image by remaining in layers +RUN set -ex && \ + apk add --no-cache --virtual build-deps build-base postgresql-dev && \ + pip install --no-cache-dir psycopg2 pycrypto && \ + apk del --no-cache build-deps -# Finally, remove packages we only needed for building -RUN yum -y remove gcc cpp glibc-devel glibc-headers kernel-headers libgomp libmpc mpfr +COPY --from=node-builder /pgadmin4/web/pgadmin/static/js/generated/ /pgadmin4/pgadmin/static/js/generated/ +COPY --from=docs-builder /pgadmin4/docs/en_US/_build/html/ /pgadmin4/docs/ -# Default config options -ENV PGADMIN_DEFAULT_EMAIL contai...@pgadmin.org -ENV PGADMIN_DEFAULT_PASSWORD Conta1ner -ENV PGADMIN_ENABLE_TLS False -ENV PGADMIN_SERVER_NAME pgadmin4 +COPY ./pgadmin4/web /pgadmin4 +COPY ./pgadmin4/requirements.txt /pgadmin4 +COPY ./run_pgadmin.py /pgadmin4 +COPY ./config_distro.py /pgadmin4 -EXPOSE 80 443 +WORKDIR /pgadmin4 +ENV PYTHONPATH=/pgadmin4 -# Start the service -ENTRYPOINT ["/bin/bash", "/entry.sh"] +RUN pip install --no-cache-dir -r requirements.txt + +# Precompile and optimize python code to save time and space on startup +RUN python -O -m compileall /pgadmin4 + +COPY ./entrypoint.sh /entrypoint.sh + +VOLUME /var/lib/pgadmin +EXPOSE 8080 8443 + +ENTRYPOINT ["/entrypoint.sh"] diff --git a/pkg/docker/README b/pkg/docker/README index b2d9595e..9fed51c9 100644 --- a/pkg/docker/README +++ b/pkg/docker/README @@ -4,14 +4,16 @@ Building ======== Whilst you can just use the Dockerfile directly, it requires that various pre-configuration steps are performed, for -example, the pgAdmin web code must be copied to ./web and yarn install/yarn run bundle must be executed. -requirements.txt is also expected to be in this directory, and the pre-built docs must be in web/docs. +example, the pgAdmin web code must be copied to `./web`, Sphinx documentation source must be copied to `./docs` +and `requirements.txt` is also expected to be in this directory. The recommended (and easy) way to build the container is to do: +```console cd $PGADMIN_SRC/ workon pgadmin-venv make docker +``` This will call the build script $PGADMIN_SRC/pkg/docker/build.sh which will prepare a staging directory containing all the required files, then build the container and push it to your repo. @@ -21,57 +23,51 @@ Running The container will accept the following variables at startup: -PGADMIN_DEFAULT_EMAIL ---------------------- - -Default: contai...@pgadmin.org) +PGADMIN_SETUP_EMAIL +------------------- This is the email address used when setting up the initial administrator account to login to pgAdmin. -PGADMIN_DEFAULT_PASSWORD ------------------------- - -Default: Conta1ner +PGADMIN_SETUP_PASSWORD +---------------------- This is the password used when setting up the initial administrator account to login to pgAdmin. PGADMIN_ENABLE_TLS ------------------ -Default: False +Default: unset -If set to the default, False, the container will listen on port 80 for connections in plain text. If set to True, the -container will listen on port 443 for TLS connections. +If not set, the container will listen on port 8080 for connections in insecure HTTP protocol. +If set to any value, the container will listen on port 8443 for TLS connections. -When TLS is enabled, a certificate and key must be provided. Typically these should be stored on the host file system -and mounted from the container. The expected paths are /certs/server.crt and /certs/server.key +When TLS is enabled, a certificate and key must be provided. +Typically these should be stored on the host file system and mounted from the container. +The expected paths are `/certs/server.crt` and `/certs/server.key`. -PGADMIN_SERVER_NAME -------------------- - -Default: pgadmin4 - -This variable allows you to specify the value used for the Apache HTTPD ServerName directive. This is commonly used to -ensure the CN of the TLS certificate matches what the server expects. +You need to explicitly map these ports with `-p` option to some port at your machine. Examples ======== Run a simple container over port 80: -docker run -p 80:80 \ - -e "PGADMIN_DEFAULT_EMAIL=u...@domain.com" \ - -e "PGADMIN_DEFAULT_PASSWORD=SuperSecret" \ +```console +docker run -p 80:8080 \ + -e "PGADMIN_SETUP_EMAIL=u...@domain.com" \ + -e "PGADMIN_SETUP_PASSWORD=SuperSecret" \ -d pgadmin4 +``` Run a TLS secured container using a shared config/storage directory in /private/var/lib/pgadmin on the host: -docker run -p 443:443 \ +```console +docker run -p 443:8443 \ -v "/private/var/lib/pgadmin:/var/lib/pgadmin" \ -v "/path/to/certificate.cert:/certs/server.cert" \ -v "/path/to/certificate.key:/certs/server.key" \ - -e "PGADMIN_DEFAULT_EMAIL=u...@domain.com" \ - -e "PGADMIN_DEFAULT_PASSWORD=SuperSecret" \ - -e "PGADMIN_ENABLE_TLS=True" \ - -e "PGADMIN_SERVER_NAME=pgadmin.domain.com" \ - -d pgadmin4 \ No newline at end of file + -e "PGADMIN_SETUP_EMAIL=u...@domain.com" \ + -e "PGADMIN_SETUP_PASSWORD=SuperSecret" \ + -e "PGADMIN_ENABLE_TLS=1" \ + -d pgadmin4 +``` diff --git a/pkg/docker/build.sh b/pkg/docker/build.sh index 373c99b7..1dda3ca2 100755 --- a/pkg/docker/build.sh +++ b/pkg/docker/build.sh @@ -41,60 +41,21 @@ if [ -d docker-build ]; then rm -rf docker-build fi -mkdir docker-build - -# Create the output directory if not present -if [ ! -d dist ]; then - mkdir dist -fi +mkdir -p docker-build/pgadmin4 # Build the clean tree -for FILE in `git ls-files web` -do - echo Adding $FILE - # We use tar here to preserve the path, as Mac (for example) doesn't support cp --parents - tar cf - $FILE | (cd docker-build; tar xf -) -done - -pushd web - yarn install - yarn run bundle - - rm -rf pgadmin/static/js/generated/.cache - - for FILE in `ls -d pgadmin/static/js/generated/*` - do - echo Adding $FILE - tar cf - $FILE | (cd ../docker-build/web; tar xf -) - done -popd - -# Build the docs -if [ -d docs/en_US/_build/html ]; then - rm -rf docs/en_US/_build/html -fi - -LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 make -C docs/en_US -f Makefile.sphinx html - -mkdir docker-build/web/docs -cp -R docs/en_US/_build/html/* docker-build/web/docs/ - -# Configure pgAdmin -echo "HELP_PATH = '../../docs/'" >> docker-build/web/config_distro.py -echo "DEFAULT_BINARY_PATHS = {" >> docker-build/web/config_distro.py -echo " 'pg': ''," >> docker-build/web/config_distro.py -echo " 'ppas': ''," >> docker-build/web/config_distro.py -echo " 'gpdb': ''" >> docker-build/web/config_distro.py -echo "}" >> docker-build/web/config_distro.py +echo Copying source tree... +git archive HEAD -- docs web requirements.txt | tar xvf - -C docker-build/pgadmin4 # Copy the Docker specific assets into place -cp pkg/docker/Dockerfile docker-build/ -cp pkg/docker/entry.sh docker-build/ -cp pkg/docker/pgadmin4.conf.j2 docker-build/ -cp requirements.txt docker-build/ +cp pkg/docker/Dockerfile \ + pkg/docker/entrypoint.sh \ + pkg/docker/config_distro.py \ + pkg/docker/run_pgadmin.py \ + docker-build/ # Build the container docker build docker-build -t $CONTAINER_NAME \ -t $CONTAINER_NAME:latest \ -t $CONTAINER_NAME:$APP_RELEASE \ - -t $CONTAINER_NAME:$APP_LONG_VERSION \ No newline at end of file + -t $CONTAINER_NAME:$APP_LONG_VERSION diff --git a/pkg/docker/config_distro.py b/pkg/docker/config_distro.py new file mode 100644 index 00000000..71ad3c74 --- /dev/null +++ b/pkg/docker/config_distro.py @@ -0,0 +1,4 @@ +HELP_PATH = '../../docs' +DEFAULT_BINARY_PATHS = { + 'pg': '/usr/bin' +} diff --git a/pkg/docker/entry.sh b/pkg/docker/entry.sh deleted file mode 100644 index b6ba42e9..00000000 --- a/pkg/docker/entry.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/usr/bin/env bash - -######################################################################## -# -# pgAdmin 4 - PostgreSQL Tools -# -# Copyright (C) 2013 - 2018, The pgAdmin Development Team -# This software is released under the PostgreSQL Licence -# -######################################################################### - -export PGADMIN_SETUP_EMAIL=${PGADMIN_DEFAULT_EMAIL} -export PGADMIN_SETUP_PASSWORD=${PGADMIN_DEFAULT_PASSWORD} - -if [ ${PGADMIN_ENABLE_TLS} != "True" ]; then - if [ -f /etc/httpd/conf.d/ssl.conf ]; then - mv /etc/httpd/conf.d/ssl.conf /etc/httpd/conf.d/ssl.conf.disabled - fi -else - if [ -f /etc/httpd/conf.d/ssl.conf.disabled ]; then - mv /etc/httpd/conf.d/ssl.conf.disabled /etc/httpd/conf.d/ssl.conf - fi -fi - -j2 /templates/pgadmin4.conf.j2 > /etc/httpd/conf.d/pgadmin4.conf - -rm -f /run/httpd/httpd.pid - -/usr/sbin/httpd -D FOREGROUND diff --git a/pkg/docker/entrypoint.sh b/pkg/docker/entrypoint.sh new file mode 100755 index 00000000..09fa3ab0 --- /dev/null +++ b/pkg/docker/entrypoint.sh @@ -0,0 +1,21 @@ +#!/bin/sh + +if [ ! -f /var/lib/pgadmin/pgadmin4.db ]; then + if [ -z "${PGADMIN_SETUP_EMAIL}" -o -z "${PGADMIN_SETUP_PASSWORD}" ]; then + echo 'You need to specify PGADMIN_SETUP_EMAIL and PGADMIN_SETUP_PASSWORD environment variables' + exit 1 + fi + + # Initialize DB before starting Gunicorn + # Importing pgadmin4 (from this script) is enough + python run_pgadmin.py +fi + +# NOTE: currently pgadmin can run only with 1 worker due to sessions implementation +# Using --threads to have multi-threaded single-process worker + +if [ ! -z ${PGADMIN_ENABLE_TLS} ]; then + exec gunicorn --bind 0.0.0.0:8443 -w 1 --threads ${GUNICORN_THREADS:-25} --access-logfile - --keyfile /certs/server.key --certfile /certs/server.cert run_pgadmin:app +else + exec gunicorn --bind 0.0.0.0:8080 -w 1 --threads ${GUNICORN_THREADS:-25} --access-logfile - run_pgadmin:app +fi diff --git a/pkg/docker/pgadmin4.conf.j2 b/pkg/docker/pgadmin4.conf.j2 deleted file mode 100644 index d44de2d5..00000000 --- a/pkg/docker/pgadmin4.conf.j2 +++ /dev/null @@ -1,43 +0,0 @@ -######################################################################## -# -# pgAdmin 4 - PostgreSQL Tools -# -# Copyright (C) 2013 - 2018, The pgAdmin Development Team -# This software is released under the PostgreSQL Licence -# -######################################################################### - -ServerName {{ PGADMIN_SERVER_NAME }} -{% if PGADMIN_ENABLE_TLS|default('False') == 'True' %} -LoadModule ssl_module modules/mod_ssl.so - -<VirtualHost *:443> - SSLEngine on - SSLCipherSuite HIGH:!aNULL:!MD5 - SSLCertificateFile "/certs/server.cert" - SSLCertificateKeyFile "/certs/server.key" - - ServerName {{ PGADMIN_SERVER_NAME }} - WSGIDaemonProcess pgadmin processes=1 threads=25 - WSGIScriptAlias / /var/www/pgadmin/pgAdmin4.wsgi - - <Directory /var/www/pgadmin> - WSGIProcessGroup pgadmin - WSGIApplicationGroup %{GLOBAL} - Order deny,allow - Allow from all - </Directory> -</VirtualHost> -{% else %} -<VirtualHost *:80> - WSGIDaemonProcess pgadmin processes=1 threads=25 - WSGIScriptAlias / /var/www/pgadmin/pgAdmin4.wsgi - - <Directory /var/www/pgadmin> - WSGIProcessGroup pgadmin - WSGIApplicationGroup %{GLOBAL} - Order deny,allow - Allow from all - </Directory> -</VirtualHost> -{% endif %} \ No newline at end of file diff --git a/pkg/docker/run_pgadmin.py b/pkg/docker/run_pgadmin.py new file mode 100644 index 00000000..48b17735 --- /dev/null +++ b/pkg/docker/run_pgadmin.py @@ -0,0 +1,4 @@ +import builtins +builtins.SERVER_MODE = True + +from pgAdmin4 import app -- 2.16.2
