This is an automated email from the ASF dual-hosted git repository.
maximebeauchemin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-superset.git
The following commit(s) were added to refs/heads/master by this push:
new 02aa3c6 Improve development experience with Docker (#5966)
02aa3c6 is described below
commit 02aa3c6395a381e59dced848df55cb1daf733e95
Author: Victor Noël <victorn...@users.noreply.github.com>
AuthorDate: Tue Nov 27 20:19:55 2018 +0100
Improve development experience with Docker (#5966)
- Improve Docker image
- smaller
- faster to build
- deterministict dependencies (see #5958)
- Rework process to simplify setting things up
- updated documentation
- less commands to type
- no files to move and modify
- optional loading of samples
- Still working in standalone mode (without volumes for superset)
---
.dockerignore | 23 ++++++++++++
.gitignore | 7 ----
CONTRIBUTING.md | 4 +-
contrib/docker/.env | 1 +
contrib/docker/Dockerfile | 73 ++++++++++++++++++-------------------
contrib/docker/docker-build.sh | 5 ---
contrib/docker/docker-compose.yml | 23 +++++++-----
contrib/docker/docker-entrypoint.sh | 9 ++++-
contrib/docker/docker-init.sh | 15 ++------
docs/installation.rst | 25 ++++++++-----
10 files changed, 102 insertions(+), 83 deletions(-)
diff --git a/.dockerignore b/.dockerignore
new file mode 100644
index 0000000..96f42fe
--- /dev/null
+++ b/.dockerignore
@@ -0,0 +1,23 @@
+**/__pycache__/
+**/.mypy_cache
+**/.pytest_cache
+**/.tox
+**/.vscode
+**/.idea
+**/.coverage
+**/.DS_Store
+**/.eggs
+**/.python-version
+**/*.egg-info
+**/*.bak
+**/*.db
+**/*.pyc
+**/*.sqllite
+**/*.swp
+
+tests/
+docs/
+install/
+superset/assets/node_modules/
+superset/assets/cypress/
+superset/assets/coverage/
diff --git a/.gitignore b/.gitignore
index ae6f75e..c6cc07b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -44,10 +44,3 @@ yarn-error.log
*.iml
venv
@eaDir/
-
-# docker
-/Dockerfile
-/docker-build.sh
-/docker-compose.yml
-/docker-entrypoint.sh
-/docker-init.sh
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 0386248..e2c6802 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -255,11 +255,11 @@ Install third-party dependencies listed in `package.json`:
# From the root of the repository
cd superset/assets
-# Install yarn, a replacement for `npm install`
+# If needed, install yarn, a replacement for `npm install`
npm install -g yarn
# Install dependencies
-yarn install
+yarn
```
Finally, to compile frontend assets, run any of the following commands.
diff --git a/contrib/docker/.env b/contrib/docker/.env
new file mode 100644
index 0000000..0a1d068
--- /dev/null
+++ b/contrib/docker/.env
@@ -0,0 +1 @@
+COMPOSE_PROJECT_NAME=superset
diff --git a/contrib/docker/Dockerfile b/contrib/docker/Dockerfile
index 0debe1e..a59a3ed 100644
--- a/contrib/docker/Dockerfile
+++ b/contrib/docker/Dockerfile
@@ -1,64 +1,63 @@
FROM python:3.6
-MAINTAINER Xiao Hanyu <hanyu.x...@shopeemobile.com>
-
-# Add a normal user
-RUN useradd --user-group --create-home --shell /bin/bash work
+RUN useradd --user-group --create-home --no-log-init --shell /bin/bash superset
# Configure environment
ENV LANG=C.UTF-8 \
- LC_ALL=C.UTF-8 \
- HOME=/home/work
+ LC_ALL=C.UTF-8
RUN apt-get update -y
-#Install dependencies to fix `curl https support error` and `elaying package
configuration warning`
+
+# Install dependencies to fix `curl https support error` and `elaying package
configuration warning`
RUN apt-get install -y apt-transport-https apt-utils
-# Install some dependencies
-# http://airbnb.io/superset/installation.html#os-dependencies
-RUN apt-get update -y && apt-get install -y build-essential libssl-dev \
+
+# Install superset dependencies
+# https://superset.incubator.apache.org/installation.html#os-dependencies
+RUN apt-get install -y build-essential libssl-dev \
libffi-dev python3-dev libsasl2-dev libldap2-dev libxi-dev
+# Install extra useful tool for development
RUN apt-get install -y vim less postgresql-client redis-tools
# Install nodejs for custom build
-#
https://github.com/apache/incubator-superset/blob/master/docs/installation.rst#making-your-own-build
+# https://superset.incubator.apache.org/installation.html#making-your-own-build
# https://nodejs.org/en/download/package-manager/
-RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
-RUN apt-get install -y nodejs
-RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -; \
- echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee
/etc/apt/sources.list.d/yarn.list; \
- apt-get update; \
- apt-get install -y yarn
+RUN curl -sL https://deb.nodesource.com/setup_10.x | bash - \
+ && apt-get install -y nodejs
-RUN mkdir $HOME/incubator-superset
+# https://yarnpkg.com/lang/en/docs/install/#debian-stable
+RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
+ && echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee
/etc/apt/sources.list.d/yarn.list \
+ && apt-get update \
+ && apt-get install -y yarn
-WORKDIR $HOME/incubator-superset
+WORKDIR /home/superset
-COPY ./ ./
+COPY requirements.txt .
+COPY requirements-dev.txt .
-RUN mkdir -p /home/work/.cache
-RUN pip install --upgrade setuptools pip
-RUN pip install -r requirements.txt
-RUN pip install -r requirements-dev.txt
-RUN pip install -e .
+RUN pip install --upgrade setuptools pip \
+ && pip install -r requirements.txt -r requirements-dev.txt \
+ && rm -rf /root/.cache/pip
-ENV PATH=/home/work/incubator-superset/superset/bin:$PATH \
- PYTHONPATH=./superset/:$PYTHONPATH
+USER superset
-COPY docker-entrypoint.sh /usr/local/bin/
-RUN chmod +x /usr/local/bin/docker-entrypoint.sh
-RUN ln -s usr/local/bin/docker-entrypoint.sh /entrypoint.sh # backwards compat
+COPY --chown=superset:superset superset superset
-COPY ./superset ./superset
-RUN chown -R work:work $HOME
+ENV PATH=/home/superset/superset/bin:$PATH \
+ PYTHONPATH=/home/superset/superset/:$PYTHONPATH
-USER work
+RUN cd superset/assets \
+ && yarn --non-interactive --frozen-lockfile --link-duplicates \
+ && yarn run sync-backend \
+ && yarn run build \
+ && rm -rf node_modules \
+ && yarn cache clean
-RUN cd superset/assets && yarn
-RUN cd superset/assets && npm run build
+COPY contrib/docker/docker-init.sh .
+COPY contrib/docker/docker-entrypoint.sh /entrypoint.sh
+ENTRYPOINT ["/entrypoint.sh"]
HEALTHCHECK CMD ["curl", "-f", "http://localhost:8088/health";]
-ENTRYPOINT ["docker-entrypoint.sh"]
-
EXPOSE 8088
diff --git a/contrib/docker/docker-build.sh b/contrib/docker/docker-build.sh
deleted file mode 100644
index 55f7327..0000000
--- a/contrib/docker/docker-build.sh
+++ /dev/null
@@ -1,5 +0,0 @@
-#!/usr/bin/env bash
-
-set -ex
-
-docker build -t apache/incubator-superset -f Dockerfile .
diff --git a/contrib/docker/docker-compose.yml
b/contrib/docker/docker-compose.yml
index 9085793..4911905 100644
--- a/contrib/docker/docker-compose.yml
+++ b/contrib/docker/docker-compose.yml
@@ -2,14 +2,14 @@ version: '3'
services:
redis:
image: redis:3.2
- restart: always
+ restart: unless-stopped
ports:
- 6379:6379
volumes:
- redis:/data
postgres:
image: postgres:10
- restart: always
+ restart: unless-stopped
environment:
POSTGRES_DB: superset
POSTGRES_PASSWORD: superset
@@ -19,8 +19,10 @@ services:
volumes:
- postgres:/var/lib/postgresql/data
superset:
- image: apache/incubator-superset
- restart: always
+ build:
+ context: ../../
+ dockerfile: contrib/docker/Dockerfile
+ restart: unless-stopped
environment:
POSTGRES_DB: superset
POSTGRES_USER: superset
@@ -29,20 +31,21 @@ services:
POSTGRES_PORT: 5432
REDIS_HOST: redis
REDIS_PORT: 6379
- SUPERSET_ENV: local
+ # If using production, comment development volume below
+ #SUPERSET_ENV: production
+ SUPERSET_ENV: development
ports:
- 8088:8088
- command: "tail -f /dev/null"
depends_on:
- postgres
- redis
volumes:
- - .:/home/work/incubator-superset
- -
superset-node-modules:/home/work/incubator-superset/superset/assets/node_modules
+ # this is needed to communicate with the postgres and redis services
+ - ./superset_config.py:/home/superset/superset/superset_config.py
+ # this is needed for development, remove with SUPERSET_ENV=production
+ - ../../superset:/home/superset/superset
volumes:
postgres:
external: false
redis:
external: false
- superset-node-modules:
- external: false
diff --git a/contrib/docker/docker-entrypoint.sh
b/contrib/docker/docker-entrypoint.sh
old mode 100644
new mode 100755
index 3e1de5e..35ae37f
--- a/contrib/docker/docker-entrypoint.sh
+++ b/contrib/docker/docker-entrypoint.sh
@@ -3,9 +3,14 @@ set -ex
if [ "$#" -ne 0 ]; then
exec "$@"
-elif [ "$SUPERSET_ENV" = "local" ]; then
- flask run -p 8088 --with-threads --reload --debugger --host=0.0.0.0
+elif [ "$SUPERSET_ENV" = "development" ]; then
+ superset worker &
+ # needed by superset runserver
+ (cd superset/assets/ && yarn && yarn run sync-backend)
+ (cd superset/assets/ && yarn run dev) &
+ flask run -p 8088 --with-threads --reload --debugger --host=0.0.0.0
elif [ "$SUPERSET_ENV" = "production" ]; then
+ superset worker &
superset runserver -a 0.0.0.0 -w $((2 * $(getconf _NPROCESSORS_ONLN) + 1))
else
superset --help
diff --git a/contrib/docker/docker-init.sh b/contrib/docker/docker-init.sh
old mode 100644
new mode 100755
index f50046b..f92c0f5
--- a/contrib/docker/docker-init.sh
+++ b/contrib/docker/docker-init.sh
@@ -8,17 +8,10 @@ fabmanager create-admin --app superset
# Initialize the database
superset db upgrade
-# Load some data to play with
-superset load_examples
+if [ "$SUPERSET_LOAD_EXAMPLES" = "yes" ]; then
+ # Load some data to play with
+ superset load_examples
+fi
# Create default roles and permissions
superset init
-
-# Need to run `npm run build` when enter contains for first time
-cd superset/assets && npm run build && cd ../../
-
-# Start superset worker for SQL Lab
-superset worker &
-
-# Start the dev web server
-flask run -p 8088 --with-threads --reload --debugger --host=0.0.0.0
diff --git a/docs/installation.rst b/docs/installation.rst
index d5c63c4..295a20d 100644
--- a/docs/installation.rst
+++ b/docs/installation.rst
@@ -43,22 +43,29 @@ If you know docker, then you're lucky, we have shortcut
road for you to
initialize development environment: ::
git clone https://github.com/apache/incubator-superset/
- cd incubator-superset
- cp
contrib/docker/{docker-build.sh,docker-compose.yml,docker-entrypoint.sh,docker-init.sh,Dockerfile}
.
- cp contrib/docker/superset_config.py superset/
- bash -x docker-build.sh
- docker-compose up -d
- docker-compose exec superset bash
- bash docker-init.sh
+ cd incubator-superset/contrib/docker
+ # prefix with SUPERSET_LOAD_EXAMPLES=yes to load examples:
+ docker-compose run --rm superset ./docker-init.sh
+ # you can run this command everytime you need to start superset now:
+ docker-compose up
After several minutes for superset initialization to finish, you can open
a browser and view `http://localhost:8088` to start your journey.
+From there, the container server will reload on modification of the superset
python
+and javascript source code.
+Don't forget to reload the page to take the new frontend into account though.
+
+See also `CONTRIBUTING.md
<https://github.com/apache/incubator-superset/blob/master/CONTRIBUTING.md#webpack-dev-server>`_,
+for alternative way of serving the frontend.
+
+It is also possible to run Superset in non-development mode: in the
`docker-compose.yml` file remove
+the volumes needed for development and change the variable `SUPERSET_ENV` to
`production`.
+
If you are attempting to build on a Mac and it exits with 137 you need to
increase your docker resources.
OSX instructions: https://docs.docker.com/docker-for-mac/#advanced (Search for
memory)
-Or if you're curious and want to install superset from bottom up, then go
-ahead.
+Or if you're curious and want to install superset from bottom up, then go
ahead.
OS dependencies
---------------
