[
https://issues.apache.org/jira/browse/GOBBLIN-1317?focusedWorklogId=515645&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515645
]
ASF GitHub Bot logged work on GOBBLIN-1317:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 23/Nov/20 16:30
Start Date: 23/Nov/20 16:30
Worklog Time Spent: 10m
Work Description: hanghangliu commented on a change in pull request #3154:
URL: https://github.com/apache/incubator-gobblin/pull/3154#discussion_r528835914
##########
File path: gobblin-docs/user-guide/Docker-Integration.md
##########
@@ -18,68 +18,68 @@ The `gobblin/gobblin-wikipedia` repository contains images
that run the Gobblin
The `gobblin/gobblin-standalone` repository contains images that run a
[Gobblin standalone service](Gobblin-Deployment#standalone-architecture) inside
a Docker container. These images provide an easy and simple way to setup a
Gobblin standalone service on any Docker compatible machine.
+The `gobblin/gobblin-service` repository contains images that run [Gobblin as
a
service](Building-Gobblin-as-a-Service#running-gobblin-as-a-service-with-docker),
which is a service that takes in a user request (a logical flow) and converts
it into a series of Gobblin Jobs, and monitors these jobs in a distributed
manner.
+
The `gobblin/gobblin-base` and `gobblin/gobblin-distributions` repositories
are for internal use only, and are primarily useful for Gobblin developers.
-## Gobblin-Wikipedia Repository
+# Run Gobblin Standalone
+
+The Docker images for this repository can be found on Docker Hub
[here](https://hub.docker.com/r/gobblin/gobblin-standalone/). These images run
a Gobblin standalone service inside a Docker container. The Gobblin standalone
service is a long running process that can run Gobblin jobs defined in a `.job`
or `.pull` file. The job / pull files are submitted to the standalone service
by placing them in a directory on the local filesystem. The standalone service
monitors this directory for any new job / pull files and runs them either
immediately or on a scheduled basis (more information on how this works can be
found
[here](Working-with-Job-Configuration-Files#adding-or-changing-job-configuration-files)).
Running the Gobblin standalone service inside a Docker container allows
Gobblin to pick up job / pull files from a directory on the host filesystem,
run the job, and write the output back the host filesystem. All the heavy
lifting is done inside a Docker container, the user just needs to worry about
defining and submitting job / pull files. The goal is to provide a easy to
setup environment for the Gobblin standalone service.
-The Docker images for this repository can be found on Docker Hub
[here](https://hub.docker.com/r/gobblin/gobblin-wikipedia/). These images are
mainly meant to act as a "Hello World" example for the Gobblin-Docker
integration, and to provide a sanity check to see if the Gobblin-Docker
integration is working on a given machine. The image contains the Gobblin
configuration files to run the [Gobblin Wikipedia job](../Getting-Started).
When a container is launched using the `gobblin-wikipedia` image, Gobblin
starts up, runs the Wikipedia example, and then exits.
+### Set working directory
-Running the `gobblin-wikipedia` image requires taking following steps (lets
assume we want to an Ubuntu based image):
+Before running docker containers, set a working directory for Gobblin jobs:
-* Download the images from the `gobblin/gobblin-wikipedia` repository
+`export LOCAL_JOB_DIR=<local_gobblin_directory>`
-```
-docker pull gobblin/gobblin-wikipedia:ubuntu-gobblin-latest
-```
+We will use this directory as the
[volume](https://docs.docker.com/storage/volumes/) for Gobblin jobs and
outputs. Make sure your Docker has the
[access](https://docs.docker.com/docker-for-mac/#file-sharing) to this folder.
This is the prerequisite for all following example jobs.
-* Run the `gobblin/gobblin-wikipedia:ubuntu-gobblin-latest` image in a Docker
container
+### Run the docker image with simple wikipedia jobs
-```
-docker run gobblin/gobblin-wikipedia:ubuntu-gobblin-latest
-```
+Run these commands to start the docker image:
-The logs are printed to the console, and no errors should pop up. This should
provide a nice sanity check to ensure that everything is working as expected.
The output of the job will be written to a directory inside the container. When
the container exits that data will be lost. In order to preserve the output of
the job, continue to the next step.
+`docker pull gobblin/gobblin-standalone:alpine-gaas-latest`
-* Preserving the output of a Docker container requires using a [data
volume](https://docs.docker.com/engine/tutorials/dockervolumes/). To do this,
run the below command:
+`docker run gobblin/gobblin-standalone:alpine-gaas-latest`
Review comment:
Agree for the docker volume setup and changed as you proposed.
The uncaught exception was newly introduced by some recent PR. It's not only
happening in docker run, but also in direct run from local build. I have raised
the discussion in slack. Looks like it was introduced by dependency issue with
Jackson.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 515645)
Time Spent: 1h 40m (was: 1.5h)
> Add Docker Recipes and Documentations
> -------------------------------------
>
> Key: GOBBLIN-1317
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1317
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Hanghang Liu
> Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Add documentation to run Gobblin on docker end to end, with latest version.
> Add docker recipes including example Wikipedias job, from/to Kafka and HDFS
> ingestion. Add guidance for these recipes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
