This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-hop.git
The following commit(s) were added to refs/heads/master by this push:
new a944738 HOP-2497 - Add custom entry point extension option (#603)
a944738 is described below
commit a94473817da45affc0740276f0b279514c6e641b
Author: Diethard Steiner <[email protected]>
AuthorDate: Fri Feb 5 13:11:53 2021 +0000
HOP-2497 - Add custom entry point extension option (#603)
* HOP-2497 - Add custom entry point extension option - implemented
---
docker/Dockerfile | 3 ++
docker/README.md | 78 +++++++++++++++++++++++++++++++++++-
docker/resources/load-and-execute.sh | 7 ++++
3 files changed, 87 insertions(+), 1 deletion(-)
diff --git a/docker/Dockerfile b/docker/Dockerfile
index e97bc67..08e57a2 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -52,6 +52,9 @@ ENV HOP_RUN_PARAMETERS=
# The “-XX:+AggressiveHeap” tells the container to use all memory assigned to
the container.
# this removed the need to calculate the necessary heap Xmx
ENV HOP_OPTIONS=-XX:+AggressiveHeap
+# Path to custom entrypoint extension script file - optional
+# e.g. to fetch Hop project files from S3 or gitlab
+ENV HOP_CUSTOM_ENTRYPOINT_EXTENSION_SHELL_FILE_PATH=
# Define en_US.
# ENV LANGUAGE en_US.UTF-8
diff --git a/docker/README.md b/docker/README.md
index 794b356..fe65c18 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -25,7 +25,7 @@ A **Hop Docker image** supporting both **short-lived** and
**long-lived** setups
Directory | Description
--- |---
-`/opt/project-hop` | location of the hop package
+`/opt/hop` | location of the hop package
`/files` | here you should mount a directory that contains the **hop and
project config** as well as the **workflows and pipelines**.
## Environment Variables
@@ -50,6 +50,7 @@ Environment Variable | Required | Description
`HOP_SHARED_JDBC_DIRECTORY` | No | Path to the directory where the JDCB
drivers are located
`HOP_SERVER_USER` | No | Username for hop-server, only valid in
long-lived containers. Default `cluster`
`HOP_SERVER_PASS` | No | Password for hop-server user, only valid in
long-lived containers. Default `cluster`
+`HOP_CUSTOM_ENTRYPOINT_EXTENSION_SHELL_FILE_PATH` | No | Path to custom
entrypoint extension script file, e.g. to fetch Hop project files from S3 or
gitlab.
The `Required` column relates to running a short-lived container.
@@ -95,6 +96,81 @@ docker run -it --rm \
You can then access the hop-server UI from your dockerhost at
`http://localhost:8080`
+## Custom Entrypoint Extension Shell Script
+
+To make the Hop Docker image even more flexible, we added a
`HOP_CUSTOM_ENTRYPOINT_EXTENSION_SHELL_FILE_PATH` variable that accepts a path
to a custom shell script (that you provide). This shell script will run when
you start the container before your Hop project is registered with the
container's Hop config and before your Hop workflow or pipeline gets kicked off.
+This feature might come in handy when you want to run some custom logic
upfront, e.g. source Hop project files from S3 or clone them from GitHub.
+
+The custom shell file can be provided in several ways (this is not a full
list):
+
+- via the mount point (`/files`)
+- You create your own Dockerfile, define this image as the base and then use
the `COPY` instruction to copy your custom shell file in your Docker image.
+
+For the last scenario mentioned, it could be something like this:
+
+We create a simple **bash script** called `clone-git-repo.sh` in a sub-folder
called `resources`:
+
+```shell
+#!/bin/bash
+cd /home/hop
+git clone ${GIT_REPO_URI}
+chown -R hop:hop /home/hop/${GIT_REPO_NAME}
+```
+
+We also make it parameter-driven, so it any other team can use it. We create
our custom Dockerfile like so:
+
+```dockerfile
+FROM apache/incubator-hop:0.70-SNAPSHOT
+ENV GIT_REPO_URI=
+# example value:
https://github.com/diethardsteiner/apache-hop-minimal-project.git
+ENV GIT_REPO_NAME=
+# example value: apache-hop-minimal-project
+USER root
+RUN apk update \
+ && apk add --no-cache git
+# copy custom entrypoint extension shell script
+COPY --chown=hop:hop ./resources/clone-git-repo.sh /home/hop/clone-git-repo.sh
+USER hop
+```
+
+Note that apart from defining the new environment variables (that go in line
with the parameters we defined in the `clone-git-repo.sh` earlier on ), we also
`COPY` the `clone-git-repo.sh` file to user hop's home directory.
+
+Next let's build a small script which builds our custom image and then tests
it by spinning up a container and running a workflow:
+
+```shell
+#!/bin/zsh
+
+DOCKER_IMG_CHECK=$(docker images | grep ds/custom-hop)
+
+if [ ! -z "${DOCKER_IMG_CHECK}" ]; then
+ echo "removing existing ds/custom-hop image"
+ docker rmi ds/custom-hop:latest
+fi
+
+docker build . -f custom.Dockerfile -t ds/custom-hop:latest
+
+echo " ==== TESTING ====="
+
+
+HOP_DOCKER_IMAGE=ds/custom-hop:latest
+PROJECT_DEPLOYMENT_DIR=/home/hop/apache-hop-minimal-project
+
+docker run -it --rm \
+ --env HOP_LOG_LEVEL=Basic \
+ --env HOP_FILE_PATH='${PROJECT_HOME}/main.hwf' \
+ --env HOP_PROJECT_DIRECTORY=${PROJECT_DEPLOYMENT_DIR} \
+ --env HOP_PROJECT_NAME=apache-hop-minimum-project \
+ --env HOP_ENVIRONMENT_NAME=dev \
+ --env
HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS=${PROJECT_DEPLOYMENT_DIR}/dev-config.json
\
+ --env HOP_RUN_CONFIG=local \
+ --env
HOP_CUSTOM_ENTRYPOINT_EXTENSION_SHELL_FILE_PATH=/home/hop/clone-git-repo.sh \
+ --env
GIT_REPO_URI=https://github.com/diethardsteiner/apache-hop-minimal-project.git \
+ --env GIT_REPO_NAME=apache-hop-minimal-project \
+ --name my-simple-hop-container \
+ ${HOP_DOCKER_IMAGE}
+```
+
+
# Shortcomings
Currently the `hop-server` support is minimal.
diff --git a/docker/resources/load-and-execute.sh
b/docker/resources/load-and-execute.sh
index 839906f..3352c30 100755
--- a/docker/resources/load-and-execute.sh
+++ b/docker/resources/load-and-execute.sh
@@ -53,6 +53,13 @@ write_server_config() {
# retrieve files from volume
# ... done via Dockerfile via specifying a volume ...
+# allow customisation
+# e.g. to fetch hop project files from S3 or github
+if test -f "${HOP_CUSTOM_ENTRYPOINT_EXTENSION_SHELL_FILE_PATH}"; then
+ echo "Sourcing custom entry point extension:
${HOP_CUSTOM_ENTRYPOINT_EXTENSION_SHELL_FILE_PATH}"
+ source ${HOP_CUSTOM_ENTRYPOINT_EXTENSION_SHELL_FILE_PATH}
+fi
+
log "Registering project config with Hop"
log "${DEPLOYMENT_PATH}/hop/hop-conf.sh --project=${HOP_PROJECT_NAME}
--project-create --project-home='${HOP_PROJECT_DIRECTORY}'
--project-config-file='${HOP_PROJECT_CONFIG_FILE_NAME}'"
