This is an automated email from the ASF dual-hosted git repository.
abti pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-gobblin.git
The following commit(s) were added to refs/heads/master by this push:
new d7fc475 [GOBBLIN-1346] removes old docker images, updates docs and
only tag on release
d7fc475 is described below
commit d7fc4757d3ab3785136cf051cd7b5f6561a463a6
Author: William Lo <[email protected]>
AuthorDate: Thu Dec 31 18:23:39 2020 -0800
[GOBBLIN-1346] removes old docker images, updates docs and only tag on
release
Dear Gobblin maintainers,
Please accept this PR. I understand that it will
not be reviewed until I have checked off all the
steps below!
### JIRA
- [x] My PR addresses the following [Gobblin JIRA]
(https://issues.apache.org/jira/browse/GOBBLIN/)
issues and references them in the PR title. For
example, "[GOBBLIN-XXX] My Gobblin PR"
-
https://issues.apache.org/jira/browse/GOBBLIN-1346
### Description
- [x] Here are some details about my PR, including
screenshots (if applicable):
- Remove old docker files that were no longer
fully supported
- Update documentation to point towards apache
dockerhub repository
- Change CI/CD system to move latest tag to only
releases
### Tests
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
Closes #3185 from Will-Lo/clean-up-docker
---
.dockerignore | 4 --
.github/workflows/docker_build_publish.yaml | 25 ++++++++--
gobblin-docker/gobblin-base/ubuntu/Dockerfile | 5 --
.../ubuntu-gobblin-0.10.0/Dockerfile | 12 -----
.../ubuntu-gobblin-0.7.0/Dockerfile | 12 -----
.../ubuntu-gobblin-0.8.0/Dockerfile | 12 -----
.../ubuntu-gobblin-0.9.0/Dockerfile | 12 -----
.../gobblin-recipes/kafka-hdfs/docker-compose.yml | 3 +-
.../gobblin-service/alpine-gaas-latest/Dockerfile | 27 -----------
.../alpine-gaas-latest/docker-compose.yml | 17 -------
.../alpine-gaas-latest/entrypoint.sh | 20 --------
.../alpine-gobblin-latest/Dockerfile | 27 -----------
.../ubuntu-gobblin-0.10.0/Dockerfile | 9 ----
.../ubuntu-gobblin-0.7.0/Dockerfile | 9 ----
.../ubuntu-gobblin-0.8.0/Dockerfile | 9 ----
.../ubuntu-gobblin-0.9.0/Dockerfile | 9 ----
.../gobblin-standalone/ubuntu-gobblin-latest | 1 -
.../ubuntu-gobblin-0.10.0/Dockerfile | 13 ------
.../ubuntu-gobblin-0.7.0/Dockerfile | 16 -------
.../ubuntu-gobblin-0.8.0/Dockerfile | 16 -------
.../ubuntu-gobblin-0.9.0/Dockerfile | 13 ------
.../gobblin-wikipedia/ubuntu-gobblin-latest | 1 -
.../alpine-gobblin-latest/Dockerfile} | 27 +++++++++--
.../gobblin/alpine-gobblin-latest/entrypoint.sh | 53 ++++++++++++++++++++++
gobblin-docs/user-guide/Docker-Integration.md | 32 +++++++------
25 files changed, 116 insertions(+), 268 deletions(-)
diff --git a/.dockerignore b/.dockerignore
index e130ea6..ae7e2c9 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -9,7 +9,6 @@
*.md
build/
-gobblin-docker/
gobblin-docs/
gobblin-dist/
@@ -22,6 +21,3 @@ gobblin-dist/
travis/
temp/
-
-# allow entrypoint scripts
-!/gobblin-docker/**/*.sh
diff --git a/.github/workflows/docker_build_publish.yaml
b/.github/workflows/docker_build_publish.yaml
index 4a8b6c5..4c5a432 100644
--- a/.github/workflows/docker_build_publish.yaml
+++ b/.github/workflows/docker_build_publish.yaml
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
name: Build and Publish Docker image
on:
push:
@@ -5,9 +22,9 @@ on:
branches:
- master
- # Publish `v1.2.3` tags as releases.
+ release:
tags:
- - v*
+ - 'v*'
env:
IMAGE_NAME: apache/gobblin
@@ -42,6 +59,6 @@ jobs:
- name: Build Images and Publish
uses: docker/build-push-action@v2
with:
- tags: ${{ steps.docker_meta.outputs.tags }},${{ env.IMAGE_NAME
}}:latest
+ tags: ${{ steps.docker_meta.outputs.tags }}
push: true
- file:
./gobblin-docker/gobblin-standalone/alpine-gobblin-latest/Dockerfile
\ No newline at end of file
+ file: ./gobblin-docker/gobblin/alpine-gobblin-latest/Dockerfile
\ No newline at end of file
diff --git a/gobblin-docker/gobblin-base/ubuntu/Dockerfile
b/gobblin-docker/gobblin-base/ubuntu/Dockerfile
deleted file mode 100644
index c50ac55..0000000
--- a/gobblin-docker/gobblin-base/ubuntu/Dockerfile
+++ /dev/null
@@ -1,5 +0,0 @@
-FROM ubuntu
-
-RUN apt-get update && \
- apt-get install -y default-jre && \
- apt-get install -y curl
\ No newline at end of file
diff --git
a/gobblin-docker/gobblin-distributions/ubuntu-gobblin-0.10.0/Dockerfile
b/gobblin-docker/gobblin-distributions/ubuntu-gobblin-0.10.0/Dockerfile
deleted file mode 100644
index 5f58f07..0000000
--- a/gobblin-docker/gobblin-distributions/ubuntu-gobblin-0.10.0/Dockerfile
+++ /dev/null
@@ -1,12 +0,0 @@
-FROM gobblin/gobblin-base:ubuntu
-
-ENV RELEASE_VERSION "0.10.0"
-
-RUN mkdir -p /opt/gobblin/
-WORKDIR /opt/gobblin/
-
-# Download gobblin-distribution-$RELEASE_VERSION
-RUN curl -OL --progress-bar
https://github.com/linkedin/gobblin/releases/download/gobblin_$RELEASE_VERSION/gobblin-distribution-$RELEASE_VERSION.tar.gz
-
-# Un-tar gobblin-distribution-$RELEASE_VERSION
-RUN tar -xf gobblin-distribution-$RELEASE_VERSION.tar.gz
diff --git
a/gobblin-docker/gobblin-distributions/ubuntu-gobblin-0.7.0/Dockerfile
b/gobblin-docker/gobblin-distributions/ubuntu-gobblin-0.7.0/Dockerfile
deleted file mode 100644
index e1aac81..0000000
--- a/gobblin-docker/gobblin-distributions/ubuntu-gobblin-0.7.0/Dockerfile
+++ /dev/null
@@ -1,12 +0,0 @@
-FROM gobblin/gobblin-base:ubuntu
-
-ENV RELEASE_VERSION "0.7.0"
-
-RUN mkdir -p /opt/gobblin/
-WORKDIR /opt/gobblin/
-
-# Download gobblin-distribution
-RUN curl -OL --progress-bar
https://github.com/linkedin/gobblin/releases/download/gobblin_$RELEASE_VERSION/gobblin-distribution-$RELEASE_VERSION.tar.gz
-
-# Un-tar gobblin-distribution
-RUN tar -xf gobblin-distribution-$RELEASE_VERSION.tar.gz
diff --git
a/gobblin-docker/gobblin-distributions/ubuntu-gobblin-0.8.0/Dockerfile
b/gobblin-docker/gobblin-distributions/ubuntu-gobblin-0.8.0/Dockerfile
deleted file mode 100644
index 34b9730..0000000
--- a/gobblin-docker/gobblin-distributions/ubuntu-gobblin-0.8.0/Dockerfile
+++ /dev/null
@@ -1,12 +0,0 @@
-FROM gobblin/gobblin-base:ubuntu
-
-ENV RELEASE_VERSION "0.8.0"
-
-RUN mkdir -p /opt/gobblin/
-WORKDIR /opt/gobblin/
-
-# Download gobblin-distribution-$RELEASE_VERSION
-RUN curl -OL --progress-bar
https://github.com/linkedin/gobblin/releases/download/gobblin_$RELEASE_VERSION/gobblin-distribution-$RELEASE_VERSION.tar.gz
-
-# Un-tar gobblin-distribution-$RELEASE_VERSION
-RUN tar -xf gobblin-distribution-$RELEASE_VERSION.tar.gz
diff --git
a/gobblin-docker/gobblin-distributions/ubuntu-gobblin-0.9.0/Dockerfile
b/gobblin-docker/gobblin-distributions/ubuntu-gobblin-0.9.0/Dockerfile
deleted file mode 100644
index 6bf2a2f..0000000
--- a/gobblin-docker/gobblin-distributions/ubuntu-gobblin-0.9.0/Dockerfile
+++ /dev/null
@@ -1,12 +0,0 @@
-FROM gobblin/gobblin-base:ubuntu
-
-ENV RELEASE_VERSION "0.9.0"
-
-RUN mkdir -p /opt/gobblin/
-WORKDIR /opt/gobblin/
-
-# Download gobblin-distribution-$RELEASE_VERSION
-RUN curl -OL --progress-bar
https://github.com/linkedin/gobblin/releases/download/gobblin_$RELEASE_VERSION/gobblin-distribution-$RELEASE_VERSION.tar.gz
-
-# Un-tar gobblin-distribution-$RELEASE_VERSION
-RUN tar -xf gobblin-distribution-$RELEASE_VERSION.tar.gz
diff --git a/gobblin-docker/gobblin-recipes/kafka-hdfs/docker-compose.yml
b/gobblin-docker/gobblin-recipes/kafka-hdfs/docker-compose.yml
index 2b6d752..2a4a332 100644
--- a/gobblin-docker/gobblin-recipes/kafka-hdfs/docker-compose.yml
+++ b/gobblin-docker/gobblin-recipes/kafka-hdfs/docker-compose.yml
@@ -18,7 +18,8 @@
version: '3'
services:
gobblin-standalone:
- image: apache/gobblin:latest
+ image: apache/gobblin:master
+ command: --mode "standalone"
volumes:
- "${LOCAL_JOB_DIR}:/tmp/gobblin-standalone/jobs"
zookeeper:
diff --git a/gobblin-docker/gobblin-service/alpine-gaas-latest/Dockerfile
b/gobblin-docker/gobblin-service/alpine-gaas-latest/Dockerfile
deleted file mode 100644
index 2f07611..0000000
--- a/gobblin-docker/gobblin-service/alpine-gaas-latest/Dockerfile
+++ /dev/null
@@ -1,27 +0,0 @@
-ARG GOBBLIN_ROOT=${GOBBLIN_REPO_ROOT}
-
-FROM openjdk:8-alpine as build
-WORKDIR /home/gobblin
-ARG GOBBLIN_VERSION=latest
-COPY ${GOBBLIN_ROOT} .
-RUN apk add --no-cache bash && \
- # include libgc in the image so that protobuf generation will work properly
- wget -q -O /etc/apk/keys/sgerrand.rsa.pub
https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
- wget
https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.32-r0/glibc-2.32-r0.apk
&& \
- apk add glibc-2.32-r0.apk && \
- ./gradlew :gobblin-distribution:buildDistributionTar -Pversion=latest && \
- tar -xvf apache-gobblin-incubating-bin-${GOBBLIN_VERSION}.tar.gz && \
- rm apache-gobblin-incubating-bin-${GOBBLIN_VERSION}.tar.gz
-
-FROM openjdk:8-jre-alpine
-WORKDIR /home/gobblin/
-COPY
${GOBBLIN_ROOT}/gobblin-docker/gobblin-service/alpine-gaas-latest/entrypoint.sh
./bin/entrypoint.sh
-COPY --from=build /home/gobblin/gobblin-dist .
-
-RUN apk add --no-cache bash && \
- mkdir /tmp/templateCatalog && \
- mkdir /tmp/gobblin-as-service && \
- mkdir /tmp/gobblin-as-service/logs && \
- chmod +x ./bin/entrypoint.sh
-
-CMD ["./bin/entrypoint.sh"]
\ No newline at end of file
diff --git
a/gobblin-docker/gobblin-service/alpine-gaas-latest/docker-compose.yml
b/gobblin-docker/gobblin-service/alpine-gaas-latest/docker-compose.yml
deleted file mode 100644
index 72fe9f8..0000000
--- a/gobblin-docker/gobblin-service/alpine-gaas-latest/docker-compose.yml
+++ /dev/null
@@ -1,17 +0,0 @@
-version: '3'
-services:
- gobblin-service:
- build:
- context: ../../../
- dockerfile: gobblin-docker/gobblin-service/alpine-gaas-latest/Dockerfile
- volumes:
- - "${LOCAL_DATAPACK_DIR}:/tmp/templateCatalog"
- - "${LOCAL_JOB_DIR}:/tmp/gobblin-as-service/jobs"
- ports:
- - 6956:6956
- gobblin-standalone:
- build:
- context: ../../../
- dockerfile:
gobblin-docker/gobblin-standalone/alpine-gobblin-latest/Dockerfile
- volumes:
- - "${LOCAL_JOB_DIR}:/tmp/gobblin-standalone/jobs"
diff --git a/gobblin-docker/gobblin-service/alpine-gaas-latest/entrypoint.sh
b/gobblin-docker/gobblin-service/alpine-gaas-latest/entrypoint.sh
deleted file mode 100755
index 8bbd2c8..0000000
--- a/gobblin-docker/gobblin-service/alpine-gaas-latest/entrypoint.sh
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-GOBBLIN_HOME="$(cd `dirname $0`/..; pwd)"
-
-./bin/gobblin.sh service gobblin-as-service start --log-to-stdout "$@"
diff --git a/gobblin-docker/gobblin-standalone/alpine-gobblin-latest/Dockerfile
b/gobblin-docker/gobblin-standalone/alpine-gobblin-latest/Dockerfile
deleted file mode 100644
index 65cd4b6..0000000
--- a/gobblin-docker/gobblin-standalone/alpine-gobblin-latest/Dockerfile
+++ /dev/null
@@ -1,27 +0,0 @@
-ARG GOBBLIN_ROOT=${GOBBLIN_REPO_ROOT}
-
-FROM openjdk:8-alpine as build
-WORKDIR /home/gobblin
-ARG GOBBLIN_VERSION=latest
-COPY ${GOBBLIN_ROOT} .
-RUN apk add --no-cache bash && \
- # include libgc in the image so that protobuf generation will work properly
- wget -q -O /etc/apk/keys/sgerrand.rsa.pub
https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
- wget
https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.32-r0/glibc-2.32-r0.apk
&& \
- apk add glibc-2.32-r0.apk && \
- ./gradlew :gobblin-distribution:buildDistributionTar -Pversion=latest && \
- tar -xvf apache-gobblin-incubating-bin-${GOBBLIN_VERSION}.tar.gz && \
- rm apache-gobblin-incubating-bin-${GOBBLIN_VERSION}.tar.gz
-
-FROM openjdk:8-jre-alpine
-WORKDIR /home/gobblin/
-COPY
${GOBBLIN_ROOT}/gobblin-docker/gobblin-standalone/alpine-gobblin-latest/entrypoint.sh
./bin/entrypoint.sh
-COPY --from=build /home/gobblin/gobblin-dist .
-RUN apk add --no-cache bash && \
- mkdir /tmp/gobblin-standalone && \
- mkdir /tmp/gobblin-standalone/jobs && \
- chmod +x ./bin/entrypoint.sh
-ENV GOBBLIN_WORK_DIR=/tmp/gobblin-standalone/jobs
-ENV GOBBLIN_JOB_CONFIG_DIR=/tmp/gobblin-standalone/jobs
-
-CMD ["./bin/entrypoint.sh"]
\ No newline at end of file
diff --git a/gobblin-docker/gobblin-standalone/ubuntu-gobblin-0.10.0/Dockerfile
b/gobblin-docker/gobblin-standalone/ubuntu-gobblin-0.10.0/Dockerfile
deleted file mode 100644
index 32b2f36..0000000
--- a/gobblin-docker/gobblin-standalone/ubuntu-gobblin-0.10.0/Dockerfile
+++ /dev/null
@@ -1,9 +0,0 @@
-FROM gobblin/gobblin-distributions:ubuntu-gobblin-0.10.0
-
-RUN mkdir -p /etc/opt/job-conf
-RUN mkdir -p /home/gobblin/work-dir
-
-ENV GOBBLIN_WORK_DIR /home/gobblin/work-dir
-ENV GOBBLIN_JOB_CONFIG_DIR /etc/opt/job-conf
-
-CMD ["java", "-cp", "/opt/gobblin/gobblin-dist/lib/*",
"-Dlog4j.configuration=file:/opt/gobblin/gobblin-dist/conf/log4j-standalone.xml",
"-Dgobblin.logs.dir=/var/log/gobblin",
"-Dorg.quartz.properties=/opt/gobblin/gobblin-dist/conf/quartz.properties",
"org.apache.gobblin.scheduler.SchedulerDaemon",
"/opt/gobblin/gobblin-dist/conf/gobblin-standalone.properties"]
diff --git a/gobblin-docker/gobblin-standalone/ubuntu-gobblin-0.7.0/Dockerfile
b/gobblin-docker/gobblin-standalone/ubuntu-gobblin-0.7.0/Dockerfile
deleted file mode 100644
index 8322083..0000000
--- a/gobblin-docker/gobblin-standalone/ubuntu-gobblin-0.7.0/Dockerfile
+++ /dev/null
@@ -1,9 +0,0 @@
-FROM gobblin/gobblin-distributions:ubuntu-gobblin-0.7.0
-
-RUN mkdir -p /etc/opt/job-conf
-RUN mkdir -p /home/gobblin/work-dir
-
-ENV GOBBLIN_WORK_DIR /home/gobblin/work-dir
-ENV GOBBLIN_JOB_CONFIG_DIR /etc/opt/job-conf
-
-CMD ["java", "-cp", "/opt/gobblin/gobblin-dist/lib/*",
"-Dlog4j.configuration=file:/opt/gobblin/gobblin-dist/conf/log4j-standalone.xml",
"-Dgobblin.logs.dir=/var/log/gobblin",
"-Dorg.quartz.properties=/opt/gobblin/gobblin-dist/conf/quartz.properties",
"org.apache.gobblin.scheduler.SchedulerDaemon",
"/opt/gobblin/gobblin-dist/conf/gobblin-standalone.properties"]
diff --git a/gobblin-docker/gobblin-standalone/ubuntu-gobblin-0.8.0/Dockerfile
b/gobblin-docker/gobblin-standalone/ubuntu-gobblin-0.8.0/Dockerfile
deleted file mode 100644
index 949b5ba..0000000
--- a/gobblin-docker/gobblin-standalone/ubuntu-gobblin-0.8.0/Dockerfile
+++ /dev/null
@@ -1,9 +0,0 @@
-FROM gobblin/gobblin-distributions:ubuntu-gobblin-0.8.0
-
-RUN mkdir -p /etc/opt/job-conf
-RUN mkdir -p /home/gobblin/work-dir
-
-ENV GOBBLIN_WORK_DIR /home/gobblin/work-dir
-ENV GOBBLIN_JOB_CONFIG_DIR /etc/opt/job-conf
-
-CMD ["java", "-cp", "/opt/gobblin/gobblin-dist/lib/*",
"-Dlog4j.configuration=file:/opt/gobblin/gobblin-dist/conf/log4j-standalone.xml",
"-Dgobblin.logs.dir=/var/log/gobblin",
"-Dorg.quartz.properties=/opt/gobblin/gobblin-dist/conf/quartz.properties",
"org.apache.gobblin.scheduler.SchedulerDaemon",
"/opt/gobblin/gobblin-dist/conf/gobblin-standalone.properties"]
diff --git a/gobblin-docker/gobblin-standalone/ubuntu-gobblin-0.9.0/Dockerfile
b/gobblin-docker/gobblin-standalone/ubuntu-gobblin-0.9.0/Dockerfile
deleted file mode 100644
index bf27076..0000000
--- a/gobblin-docker/gobblin-standalone/ubuntu-gobblin-0.9.0/Dockerfile
+++ /dev/null
@@ -1,9 +0,0 @@
-FROM gobblin/gobblin-distributions:ubuntu-gobblin-0.9.0
-
-RUN mkdir -p /etc/opt/job-conf
-RUN mkdir -p /home/gobblin/work-dir
-
-ENV GOBBLIN_WORK_DIR /home/gobblin/work-dir
-ENV GOBBLIN_JOB_CONFIG_DIR /etc/opt/job-conf
-
-CMD ["java", "-cp", "/opt/gobblin/gobblin-dist/lib/*",
"-Dlog4j.configuration=file:/opt/gobblin/gobblin-dist/conf/log4j-standalone.xml",
"-Dgobblin.logs.dir=/var/log/gobblin",
"-Dorg.quartz.properties=/opt/gobblin/gobblin-dist/conf/quartz.properties",
"org.apache.gobblin.scheduler.SchedulerDaemon",
"/opt/gobblin/gobblin-dist/conf/gobblin-standalone.properties"]
diff --git a/gobblin-docker/gobblin-standalone/ubuntu-gobblin-latest
b/gobblin-docker/gobblin-standalone/ubuntu-gobblin-latest
deleted file mode 120000
index 829be1a..0000000
--- a/gobblin-docker/gobblin-standalone/ubuntu-gobblin-latest
+++ /dev/null
@@ -1 +0,0 @@
-ubuntu-gobblin-0.10.0
\ No newline at end of file
diff --git a/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-0.10.0/Dockerfile
b/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-0.10.0/Dockerfile
deleted file mode 100644
index 8f46fa5..0000000
--- a/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-0.10.0/Dockerfile
+++ /dev/null
@@ -1,13 +0,0 @@
-FROM gobblin/gobblin-distributions:ubuntu-gobblin-0.10.0
-
-ENV RELEASE_VERSION "0.10.0"
-ENV GOBBLIN_WORK_DIR /home/gobblin/work-dir
-
-RUN echo "Downloading Gobblin Wikipedia Configuration Files"
-RUN mkdir -p /etc/opt/gobblin/job-conf
-
-# Downloading wikipedia.pull
-RUN curl -L --progress-bar
https://raw.githubusercontent.com/linkedin/gobblin/gobblin_$RELEASE_VERSION/gobblin-example/src/main/resources/wikipedia.pull
-o /etc/opt/gobblin/job-conf/wikipedia.pull
-
-# Start Gobblin Wikipedia Example
-CMD ["java", "-cp", "/opt/gobblin/gobblin-dist/lib/*",
"-Dlog4j.configuration=file:/opt/gobblin/gobblin-dist/conf/log4j.properties",
"org.apache.gobblin.runtime.local.CliLocalJobLauncher", "--sysconfig",
"/opt/gobblin/gobblin-dist/conf/gobblin-cli.properties", "--jobconfig",
"/etc/opt/gobblin/job-conf/wikipedia.pull"]
diff --git a/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-0.7.0/Dockerfile
b/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-0.7.0/Dockerfile
deleted file mode 100644
index 04f5297..0000000
--- a/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-0.7.0/Dockerfile
+++ /dev/null
@@ -1,16 +0,0 @@
-FROM gobblin/gobblin-distributions:ubuntu-gobblin-0.7.0
-
-ENV RELEASE_VERSION "0.7.0"
-ENV GOBBLIN_WORK_DIR /home/gobblin/work-dir
-
-RUN echo "Downloading Gobblin Wikipedia Configuration Files"
-RUN mkdir -p /etc/opt/gobblin/job-conf
-
-# Downloading wikipedia.pull
-RUN curl -L --progress-bar
https://raw.githubusercontent.com/linkedin/gobblin/gobblin_$RELEASE_VERSION/gobblin-example/src/main/resources/wikipedia.pull
-o /etc/opt/gobblin/job-conf/wikipedia.pull
-
-# Downloading gobblin-cli.properties since its not in release 0.7.0
-RUN curl -L --progress-bar
https://raw.githubusercontent.com/linkedin/gobblin/gobblin-docker/conf/gobblin-cli.properties
-o /opt/gobblin/gobblin-dist/conf/gobblin-cli.properties
-
-# Start Gobblin Wikipedia Example
-CMD ["java", "-cp", "/opt/gobblin/gobblin-dist/lib/*",
"-Dlog4j.configuration=file:/opt/gobblin/gobblin-dist/conf/log4j.properties",
"org.apache.gobblin.runtime.local.CliLocalJobLauncher", "--sysconfig",
"/opt/gobblin/gobblin-dist/conf/gobblin-cli.properties", "--jobconfig",
"/etc/opt/gobblin/job-conf/wikipedia.pull"]
diff --git a/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-0.8.0/Dockerfile
b/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-0.8.0/Dockerfile
deleted file mode 100644
index 18a13c7..0000000
--- a/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-0.8.0/Dockerfile
+++ /dev/null
@@ -1,16 +0,0 @@
-FROM gobblin/gobblin-distributions:ubuntu-gobblin-0.8.0
-
-ENV RELEASE_VERSION "0.8.0"
-ENV GOBBLIN_WORK_DIR /home/gobblin/work-dir
-
-RUN echo "Downloading Gobblin Wikipedia Configuration Files"
-RUN mkdir -p /etc/opt/gobblin/job-conf
-
-# Downloading wikipedia.pull
-RUN curl -L --progress-bar
https://raw.githubusercontent.com/linkedin/gobblin/gobblin_$RELEASE_VERSION/gobblin-example/src/main/resources/wikipedia.pull
-o /etc/opt/gobblin/job-conf/wikipedia.pull
-
-# Downloading gobblin-cli.properties since its not in release 0.8.0
-RUN curl -L --progress-bar
https://raw.githubusercontent.com/linkedin/gobblin/gobblin-docker/conf/gobblin-cli.properties
-o /opt/gobblin/gobblin-dist/conf/gobblin-cli.properties
-
-# Start Gobblin Wikipedia Example
-CMD ["java", "-cp", "/opt/gobblin/gobblin-dist/lib/*",
"-Dlog4j.configuration=file:/opt/gobblin/gobblin-dist/conf/log4j.properties",
"org.apache.gobblin.runtime.local.CliLocalJobLauncher", "--sysconfig",
"/opt/gobblin/gobblin-dist/conf/gobblin-cli.properties", "--jobconfig",
"/etc/opt/gobblin/job-conf/wikipedia.pull"]
diff --git a/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-0.9.0/Dockerfile
b/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-0.9.0/Dockerfile
deleted file mode 100644
index 8f46c83..0000000
--- a/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-0.9.0/Dockerfile
+++ /dev/null
@@ -1,13 +0,0 @@
-FROM gobblin/gobblin-distributions:ubuntu-gobblin-0.9.0
-
-ENV RELEASE_VERSION "0.9.0"
-ENV GOBBLIN_WORK_DIR /home/gobblin/work-dir
-
-RUN echo "Downloading Gobblin Wikipedia Configuration Files"
-RUN mkdir -p /etc/opt/gobblin/job-conf
-
-# Downloading wikipedia.pull
-RUN curl -L --progress-bar
https://raw.githubusercontent.com/linkedin/gobblin/gobblin_$RELEASE_VERSION/gobblin-example/src/main/resources/wikipedia.pull
-o /etc/opt/gobblin/job-conf/wikipedia.pull
-
-# Start Gobblin Wikipedia Example
-CMD ["java", "-cp", "/opt/gobblin/gobblin-dist/lib/*",
"-Dlog4j.configuration=file:/opt/gobblin/gobblin-dist/conf/log4j.properties",
"org.apache.gobblin.runtime.local.CliLocalJobLauncher", "--sysconfig",
"/opt/gobblin/gobblin-dist/conf/gobblin-cli.properties", "--jobconfig",
"/etc/opt/gobblin/job-conf/wikipedia.pull"]
diff --git a/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-latest
b/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-latest
deleted file mode 120000
index 829be1a..0000000
--- a/gobblin-docker/gobblin-wikipedia/ubuntu-gobblin-latest
+++ /dev/null
@@ -1 +0,0 @@
-ubuntu-gobblin-0.10.0
\ No newline at end of file
diff --git
a/gobblin-docker/gobblin-standalone/alpine-gobblin-latest/entrypoint.sh
b/gobblin-docker/gobblin/alpine-gobblin-latest/Dockerfile
similarity index 50%
rename from
gobblin-docker/gobblin-standalone/alpine-gobblin-latest/entrypoint.sh
rename to gobblin-docker/gobblin/alpine-gobblin-latest/Dockerfile
index b5875da..071129f 100644
--- a/gobblin-docker/gobblin-standalone/alpine-gobblin-latest/entrypoint.sh
+++ b/gobblin-docker/gobblin/alpine-gobblin-latest/Dockerfile
@@ -1,5 +1,3 @@
-#!/bin/bash
-
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -15,6 +13,27 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-GOBBLIN_HOME="$(cd `dirname $0`/..; pwd)"
-./bin/gobblin.sh service standalone start --log-to-stdout $@
+FROM gradle:5.6.4-jdk8 as build
+WORKDIR /home/gobblin
+ARG GOBBLIN_FLAVOR=standard
+
+# copy the entire project folder
+COPY . .
+RUN gradle build -x findbugsMain -x test -x rat -x checkstyleMain -x javadoc
-x checkstyleTest -PgobblinFlavor=${GOBBLIN_FLAVOR} --no-daemon && \
+ tar -xvf ./build/gobblin-distribution/distributions/*.tar.gz && \
+ rm *.tar.gz
+
+FROM openjdk:8-jre-alpine
+WORKDIR /home/gobblin/
+COPY ./gobblin-docker/gobblin/alpine-gobblin-latest/entrypoint.sh
./bin/entrypoint.sh
+COPY --from=build /home/gobblin/gobblin-dist .
+RUN apk add --no-cache bash && \
+ mkdir /tmp/gobblin && \
+ mkdir /tmp/gobblin/jobs && \
+ chmod +x ./bin/entrypoint.sh
+
+ENV GOBBLIN_WORK_DIR=/tmp/gobblin/
+ENV GOBBLIN_JOB_CONFIG_DIR=/tmp/gobblin/jobs
+
+ENTRYPOINT ["./bin/entrypoint.sh"]
\ No newline at end of file
diff --git a/gobblin-docker/gobblin/alpine-gobblin-latest/entrypoint.sh
b/gobblin-docker/gobblin/alpine-gobblin-latest/entrypoint.sh
new file mode 100755
index 0000000..ff717b8
--- /dev/null
+++ b/gobblin-docker/gobblin/alpine-gobblin-latest/entrypoint.sh
@@ -0,0 +1,53 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Execution Modes
+
+STANDALONE_MODE='standalone'
+CLUSTER_MASTER_MODE='cluster-master'
+CLUSTER_WORKER_MODE='cluster-worker'
+AWS_MODE='aws'
+YARN_MODE='yarn'
+MAPREDUCE_MODE='mapreduce'
+GOBBLIN_AS_SERVICE_MODE='gobblin-as-service'
+
+GOBBLIN_EXEC_MODE_LIST="$STANDALONE_MODE, $CLUSTER_MASTER_MODE,
$CLUSTER_WORKER_MODE, $AWS_MODE, $YARN_MODE, $MAPREDUCE_MODE,
$GOBBLIN_AS_SERVICE_MODE"
+GOBBLIN_HOME="$(cd `dirname $0`/..; pwd)"
+
+EXECUTION_MODE=''
+CONF_PATH=''
+ARGS="$@"
+
+shopt -s nocasematch
+for i in $ARGS
+ do
+ case "$1" in
+ --mode )
+ EXECUTION_MODE="$2"
+ shift
+ ;;
+ esac
+ shift
+done
+
+if [[ -z "$EXECUTION_MODE" ]]; then
+ echo "Execution mode missing: expected one of the following:
$GOBBLIN_EXEC_MODE_LIST"
+ echo "Usage: docker run <docker_image_tag> --mode <execution_mode>
<additional_options>"
+ exit 1
+fi
+
+./bin/gobblin.sh service $EXECUTION_MODE start --log-to-stdout $ARGS
diff --git a/gobblin-docs/user-guide/Docker-Integration.md
b/gobblin-docs/user-guide/Docker-Integration.md
index f7989c1..c38020d 100644
--- a/gobblin-docs/user-guide/Docker-Integration.md
+++ b/gobblin-docs/user-guide/Docker-Integration.md
@@ -12,21 +12,25 @@ For more information on Docker, including how to install
it, check out the docum
# Docker Repositories
-Gobblin currently has four different repositories, and all are on Docker Hub
[here](https://hub.docker.com/u/gobblin/). We are also starting to use
[Apache's
repository](https://hub.docker.com/r/apache/gobblin/tags?page=1&ordering=last_updated)
for our images.
+Github Actions pushes the latest docker image to the Apache DockerHub
repository [here](https://hub.docker.com/r/apache/gobblin) from
`gobblin-docker/gobblin/alpine-gobblin-latest/Dockerfile`
-The `gobblin/gobblin-wikipedia` repository contains images that run the
Gobblin Wikipedia job found in the [getting started guide](../Getting-Started).
These images are useful for users new to Docker or Gobblin, they primarily act
as a "Hello World" example for the Gobblin Docker integration.
+To run this image, you will need to pass in the corresponding execution mode.
The execution modes can be found
[here](https://gobblin.readthedocs.io/en/latest/user-guide/Gobblin-Deployment/)
-The `gobblin/gobblin-standalone` repository contains images that run a
[Gobblin standalone service](Gobblin-Deployment#standalone-architecture) inside
a Docker container. These images provide an easy and simple way to setup a
Gobblin standalone service on any Docker compatible machine.
+```
+docker pull apache/gobblin
+docker run apache/gobblin --mode <execution mode> <additional args>
+```
-The `gobblin/gobblin-service` repository contains images that run [Gobblin as
a
service](Building-Gobblin-as-a-Service#running-gobblin-as-a-service-with-docker),
which is a service that takes in a user request (a logical flow) and converts
it into a series of Gobblin Jobs, and monitors these jobs in a distributed
manner.
+For example, to run Gobblin in standalone mode
+```
+docker run apache/gobblin --mode standalone
+```
-The `gobblin/gobblin-base` and `gobblin/gobblin-distributions` repositories
are for internal use only, and are primarily useful for Gobblin developers.
-
-# Run Gobblin Standalone
-
-The Docker images for this repository can be found on Docker Hub
[here](https://hub.docker.com/r/gobblin/gobblin-standalone/). These images run
a Gobblin standalone service inside a Docker container. The Gobblin standalone
service is a long running process that can run Gobblin jobs defined in a `.job`
or `.pull` file. The job / pull files are submitted to the standalone service
by placing them in a directory on the local filesystem. The standalone service
monitors this directory for any [...]
-
-### Set working directory
+To pass your own configuration to Gobblin standalone, use a docker volume. Due
to the nature of the startup script, the volumes
+will need to be declared before the arguments are passed to the execution
mode. E.g.
+```
+docker run -v <path to local configuration
files>:/home/gobblin/conf/standalone apache/gobblin --mode standalone
+```
Before running docker containers, set a working directory for Gobblin jobs:
@@ -40,7 +44,7 @@ Run these commands to start the docker image:
`docker pull apache/gobblin:latest`
-`docker run -v $LOCAL_JOB_DIR:/tmp/gobblin-standalone/jobs
apache/gobblin:latest`
+`docker run -v $LOCAL_JOB_DIR:/tmp/gobblin-standalone/jobs
apache/gobblin:latest --mode standalone`
After the container spins up, put the
[wikipedia.pull](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)
in ${LOCAL_JOB_DIR}. You will see the Gobblin daemon pick up the job, and the
result output is in ${LOCAL_JOB_DIR}/job-output/.
@@ -94,9 +98,7 @@ Similar to standalone working directory settings:
Run these commands to start the docker image:
-`docker pull gobblin/gobblin-service:alpine-gaas-latest`
-
-`docker run -p 6956:6956 -v GAAS_JOB_DIR:/tmp/gobblin-as-service/jobs -v
LOCAL_DATAPACK_DIR:/tmp/templateCatalog
gobblin/gobblin-service:alpine-gaas-latest`
+`docker run -p 6956:6956 -v $GAAS_JOB_DIR:/tmp/gobblin-as-service/jobs -v
$LOCAL_DATAPACK_DIR:/tmp/templateCatalog apache/gobblin --mode
gobblin-as-service`
The GaaS will be started, and the service can now be accessed on
localhost:6956.
