[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=324576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324576 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 07/Oct/19 19:57 Start Date: 07/Oct/19 19:57 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324576) Time Spent: 9h 10m (was: 9h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 9h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323726=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323726 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:47 Start Date: 04/Oct/19 20:47 Worklog Time Spent: 10m Work Description: soyrice commented on issue #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#issuecomment-538554236 > (Please ping me when the open comments are addressed.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323726) Time Spent: 8h 50m (was: 8h 40m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 8h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323723=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323723 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:47 Start Date: 04/Oct/19 20:47 Worklog Time Spent: 10m Work Description: soyrice commented on issue #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#issuecomment-538554236 > (Please ping me when the open comments are addressed.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323723) Time Spent: 8h 20m (was: 8h 10m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 8h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323724=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323724 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:47 Start Date: 04/Oct/19 20:47 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323724) Time Spent: 8.5h (was: 8h 20m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 8.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323725=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323725 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:47 Start Date: 04/Oct/19 20:47 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607 - Create new section on Beam website about runtime systems - Move execution model doc from SDK folder to this section - Add how-to guide about customizing SDK container images R: @aaltay, @rosetn Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323712 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:44 Start Date: 04/Oct/19 20:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r331608703 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323712) Time Spent: 7h 40m (was: 7.5h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 7h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323715
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 04/Oct/19 20:44
Start Date: 04/Oct/19 20:44
Worklog Time Spent: 10m
Work Description: soyrice commented on pull request #9607: [BEAM-8209]
Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r331596805
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 323715)
Time Spent: 8h (was: 7h 50m)
> Document custom docker containers
> -
>
> Key: BEAM-8209
> URL: https://issues.apache.org/jira/browse/BEAM-8209
>
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323701=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323701 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:44 Start Date: 04/Oct/19 20:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r331608624 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323701) Time Spent: 6h 40m (was: 6.5h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 6h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323706=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323706 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:44 Start Date: 04/Oct/19 20:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r331574058 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323706) Time Spent: 7h 10m (was: 7h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 7h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323709=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323709
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 04/Oct/19 20:44
Start Date: 04/Oct/19 20:44
Worklog Time Spent: 10m
Work Description: soyrice commented on pull request #9607: [BEAM-8209]
Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r331595406
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323716=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323716
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 04/Oct/19 20:44
Start Date: 04/Oct/19 20:44
Worklog Time Spent: 10m
Work Description: soyrice commented on pull request #9607: [BEAM-8209]
Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r331578481
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323703=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323703
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 04/Oct/19 20:44
Start Date: 04/Oct/19 20:44
Worklog Time Spent: 10m
Work Description: soyrice commented on pull request #9607: [BEAM-8209]
Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r331595917
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323714
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 04/Oct/19 20:44
Start Date: 04/Oct/19 20:44
Worklog Time Spent: 10m
Work Description: soyrice commented on pull request #9607: [BEAM-8209]
Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r331580157
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323711=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323711 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:44 Start Date: 04/Oct/19 20:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r331576178 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323711) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 7.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323699=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323699 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:44 Start Date: 04/Oct/19 20:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r331574020 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323699) Time Spent: 6h 20m (was: 6h 10m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 6h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323705 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:44 Start Date: 04/Oct/19 20:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r331569816 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: + +``` +./gradlew docker -Pdocker-tag=2.3.0 +``` + +To change the `docker` target, build the container with the `docker-repository-root` option: Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323705) Time Spent: 7h (was: 6h 50m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 7h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323710
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 04/Oct/19 20:44
Start Date: 04/Oct/19 20:44
Worklog Time Spent: 10m
Work Description: soyrice commented on pull request #9607: [BEAM-8209]
Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r331596572
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323702=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323702 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:44 Start Date: 04/Oct/19 20:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r331574098 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323702) Time Spent: 6h 50m (was: 6h 40m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 6h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323719=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323719
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 04/Oct/19 20:44
Start Date: 04/Oct/19 20:44
Worklog Time Spent: 10m
Work Description: soyrice commented on pull request #9607: [BEAM-8209]
Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r331590943
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323700=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323700 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:44 Start Date: 04/Oct/19 20:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r331569705 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323700) Time Spent: 6.5h (was: 6h 20m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323713 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:44 Start Date: 04/Oct/19 20:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r331576214 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323713) Time Spent: 7h 50m (was: 7h 40m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 7h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323718 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:44 Start Date: 04/Oct/19 20:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r331630456 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323718) Time Spent: 8h 10m (was: 8h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 8h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323708 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:44 Start Date: 04/Oct/19 20:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r331634126 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: + +``` +./gradlew docker -Pdocker-tag=2.3.0 Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323708) Time Spent: 7h 10m (was: 7h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 7h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323717 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 04/Oct/19 20:44 Start Date: 04/Oct/19 20:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r331571409 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 323717) Time Spent: 8h 10m (was: 8h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 8h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=323704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323704
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 04/Oct/19 20:44
Start Date: 04/Oct/19 20:44
Worklog Time Spent: 10m
Work Description: soyrice commented on pull request #9607: [BEAM-8209]
Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r331596311
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=319694=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319694 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 27/Sep/19 17:24 Start Date: 27/Sep/19 17:24 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r329174335 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: + +``` +./gradlew docker -Pdocker-tag=2.3.0 +``` + +To change the `docker` target, build the container with the `docker-repository-root` option: + +``` +./gradlew docker -Pdocker-repository-root=$LOCATION +``` + +## Customizing container images + +You can add extra dependencies or serialization files to container images so the execution engine doesn't need them. Review comment: Updated to: "You can add extra dependencies to container images so that you don't have to supply the dependencies to execution engines." This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319694) Time Spent: 6h 10m (was: 6h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=319693=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319693 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 27/Sep/19 17:24 Start Date: 27/Sep/19 17:24 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r329174335 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: + +``` +./gradlew docker -Pdocker-tag=2.3.0 +``` + +To change the `docker` target, build the container with the `docker-repository-root` option: + +``` +./gradlew docker -Pdocker-repository-root=$LOCATION +``` + +## Customizing container images + +You can add extra dependencies or serialization files to container images so the execution engine doesn't need them. Review comment: Updated to: "You can add extra dependencies to container images so you don't have to supply the dependencies to execution engines." This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319693) Time Spent: 6h (was: 5h 50m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 6h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=319654=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319654 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 27/Sep/19 16:28 Start Date: 27/Sep/19 16:28 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r329154295 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: + +``` +./gradlew docker -Pdocker-tag=2.3.0 +``` + +To change the `docker` target, build the container with the `docker-repository-root` option: + +``` +./gradlew docker -Pdocker-repository-root=$LOCATION +``` + +## Customizing container images + +You can add extra dependencies or serialization files to container images so the execution engine doesn't need them. + +To customize a container image, either: +* [Write a new](#writing-new-dockerfiles) [Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the original +* [Modify](#modifying-dockerfiles) the [original Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile) and reimage the container + +It's often easier to write a new Dockerfile, but you can customize anything, including the base OS, by modifying the original. Review comment: Done. Split it into two sentences - much more digestible This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319654) Time Spent: 5h 50m (was: 5h 40m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > >
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=319649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319649 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 27/Sep/19 16:23 Start Date: 27/Sep/19 16:23 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r329152146 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319649) Time Spent: 5h 20m (was: 5h 10m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=319650=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319650 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 27/Sep/19 16:23 Start Date: 27/Sep/19 16:23 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r329152218 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319650) Time Spent: 5.5h (was: 5h 20m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=319651=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319651 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 27/Sep/19 16:23 Start Date: 27/Sep/19 16:23 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r329152375 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: + +``` +./gradlew docker -Pdocker-tag=2.3.0 +``` + +To change the `docker` target, build the container with the `docker-repository-root` option: + +``` +./gradlew docker -Pdocker-repository-root=$LOCATION +``` + +## Customizing container images + +You can add extra dependencies or serialization files to container images so the execution engine doesn't need them. + +To customize a container image, either: Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319651) Time Spent: 5h 40m (was: 5.5h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=319646=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319646
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 27/Sep/19 16:14
Start Date: 27/Sep/19 16:14
Worklog Time Spent: 10m
Work Description: soyrice commented on pull request #9607: [BEAM-8209]
Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r329148768
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=319625=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319625
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 27/Sep/19 15:45
Start Date: 27/Sep/19 15:45
Worklog Time Spent: 10m
Work Description: soyrice commented on pull request #9607: [BEAM-8209]
Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r329136880
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
Review comment:
Done
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=319624=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319624 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 27/Sep/19 15:44 Start Date: 27/Sep/19 15:44 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r329136337 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319624) Time Spent: 4h 50m (was: 4h 40m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=319622=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319622 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 27/Sep/19 15:43 Start Date: 27/Sep/19 15:43 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r329135888 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319622) Time Spent: 4.5h (was: 4h 20m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=319623=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319623 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 27/Sep/19 15:43 Start Date: 27/Sep/19 15:43 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r329136133 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319623) Time Spent: 4h 40m (was: 4.5h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=319620=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319620 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 27/Sep/19 15:41 Start Date: 27/Sep/19 15:41 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r329135019 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: Review comment: Great catch, thanks. Moved it to the other file This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319620) Time Spent: 4h 10m (was: 4h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=318669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-318669 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 26/Sep/19 00:39 Start Date: 26/Sep/19 00:39 Worklog Time Spent: 10m Work Description: aaltay commented on issue #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#issuecomment-535278969 (Please ping me when the open comments are addressed.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 318669) Time Spent: 4h (was: 3h 50m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317939 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 25/Sep/19 00:43 Start Date: 25/Sep/19 00:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327884859 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: + +``` +./gradlew docker -Pdocker-tag=2.3.0 Review comment: Let's use one of language specific commands here, we don't encourage users to use this command. And change tag to a different one, date would be a good example. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317939) Time Spent: 3h (was: 2h 50m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317952=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317952
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 25/Sep/19 00:43
Start Date: 25/Sep/19 00:43
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9607:
[BEAM-8209] Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r327886607
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
Review comment:
We don't need to pull
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317935=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317935 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 25/Sep/19 00:43 Start Date: 25/Sep/19 00:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327883788 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. Review comment: default version and location -> default repository and tag. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317935) Time Spent: 2.5h (was: 2h 20m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317949=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317949
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 25/Sep/19 00:43
Start Date: 25/Sep/19 00:43
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9607:
[BEAM-8209] Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r327886782
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
Review comment:
The link should be `https://cloud.docker.com/u/apachebeam/repository/list`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 317949)
Time Spent: 3h 40m (was: 3.5h)
> Document custom docker containers
> -
>
> Key: BEAM-8209
>
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317946 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 25/Sep/19 00:43 Start Date: 25/Sep/19 00:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327885141 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: + +``` +./gradlew docker -Pdocker-tag=2.3.0 +``` + +To change the `docker` target, build the container with the `docker-repository-root` option: Review comment: target -> repository This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317946) Time Spent: 3h 20m (was: 3h 10m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317940 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 25/Sep/19 00:43 Start Date: 25/Sep/19 00:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327875371 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker Review comment: In order to make it consist with python command, it'd be better to change this to `./gradlew :sdks:java:container:docker` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317940) Time Spent: 3h 10m (was: 3h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317944=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317944 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 25/Sep/19 00:43 Start Date: 25/Sep/19 00:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327885258 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: + +``` +./gradlew docker -Pdocker-tag=2.3.0 +``` + +To change the `docker` target, build the container with the `docker-repository-root` option: + +``` +./gradlew docker -Pdocker-repository-root=$LOCATION Review comment: same here, let's use one of language specific commands here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317944) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317937=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317937
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 25/Sep/19 00:43
Start Date: 25/Sep/19 00:43
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9607:
[BEAM-8209] Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r327887183
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317951=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317951
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 25/Sep/19 00:43
Start Date: 25/Sep/19 00:43
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9607:
[BEAM-8209] Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r327889107
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317950=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317950
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 25/Sep/19 00:43
Start Date: 25/Sep/19 00:43
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9607:
[BEAM-8209] Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r327889215
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317947 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 25/Sep/19 00:43 Start Date: 25/Sep/19 00:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327878719 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: Review comment: default Docker registry is Docker Hub. version -> tag This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317947) Time Spent: 3.5h (was: 3h 20m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317934=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317934 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 25/Sep/19 00:43 Start Date: 25/Sep/19 00:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327873509 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. Review comment: We push images to Docker Hub with `apachebeam` repository now, not Bintray with `apache`. Users don't need to create accounts to pull images. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317934) Time Spent: 2h 20m (was: 2h 10m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317942=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317942 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 25/Sep/19 00:43 Start Date: 25/Sep/19 00:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327878351 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` Review comment: Let's use these examples. ``` REPOSITORY TAG IMAGE ID CREATED SIZE apachebeam/java_sdk latest 16ca619d489e 2 weeks ago550MB apachebeam/python2.7_sdklatest b6fb40539c29 2 weeks ago 1.78GB apachebeam/python3.5_sdklatest bae309000d09 2 weeks ago 1.85GB apachebeam/python3.6_sdklatest 42faad307d1a 2 weeks ago 1.86GB apachebeam/python3.7_sdklatest 18267df54139 2 weeks ago 1.86GB apachebeam/go_sdk latest 30cf602e9763 2 weeks ago124MB ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317942) Time Spent: 3h 10m (was: 3h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317941=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317941 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 25/Sep/19 00:43 Start Date: 25/Sep/19 00:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327874238 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker Review comment: I think it's better to remove this section, it may confuse users. Instead, guiding users directly to following section is more practical. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317941) Time Spent: 3h 10m (was: 3h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317945=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317945 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 25/Sep/19 00:43 Start Date: 25/Sep/19 00:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327875577 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker Review comment: In order to make it consist with python command, it'd be better to change this to `./gradlew :sdks:go:container:docker` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317945) Time Spent: 3h 20m (was: 3h 10m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317936=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317936
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 25/Sep/19 00:43
Start Date: 25/Sep/19 00:43
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9607:
[BEAM-8209] Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r327888412
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317938=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317938 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 25/Sep/19 00:43 Start Date: 25/Sep/19 00:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327883557 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam Review comment: This should be `index.docker.io/apachebeam`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317938) Time Spent: 3h (was: 2h 50m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317943=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317943
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 25/Sep/19 00:43
Start Date: 25/Sep/19 00:43
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9607:
[BEAM-8209] Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r327888664
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317948=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317948
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 25/Sep/19 00:43
Start Date: 25/Sep/19 00:43
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9607:
[BEAM-8209] Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r327887723
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317932=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317932
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 25/Sep/19 00:42
Start Date: 25/Sep/19 00:42
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9607:
[BEAM-8209] Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r327874981
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
Review comment:
This command would not work anymore. We should use following commands.
To create images for all versions of python: `./gradlew
:sdks:python:container buildAll`
To create an image for a certain python version: `./gradlew
:sdks:python:container:{version}:docker` where version is [py2, py35, py36,
py37].
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 317932)
Time Spent: 2h (was: 1h 50m)
> Document custom docker containers
> -
>
> Key: BEAM-8209
> URL: https://issues.apache.org/jira/browse/BEAM-8209
> Project: Beam
> Issue Type: Sub-task
> Components: website
>Reporter: Cyrus Maden
>Assignee: Cyrus Maden
>Priority: Minor
> Fix For: 2.16.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317931=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317931
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 25/Sep/19 00:42
Start Date: 25/Sep/19 00:42
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9607:
[BEAM-8209] Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r327888242
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317933=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317933 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 25/Sep/19 00:43 Start Date: 25/Sep/19 00:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327883934 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: Review comment: The purpose of using `docker-tag` is to overwrite default tag, not used to specify an older Python SDK version. Python version should be specified at gradlew command. This can be changed to something like ``` To overwrite the default image tag, we can use `docker-tag` option to pass a new tag. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317933) Time Spent: 2h 10m (was: 2h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317117 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 24/Sep/19 01:31 Start Date: 24/Sep/19 01:31 Worklog Time Spent: 10m Work Description: rosetn commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327393965 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: + +``` +./gradlew docker -Pdocker-tag=2.3.0 +``` + +To change the `docker` target, build the container with the `docker-repository-root` option: + +``` +./gradlew docker -Pdocker-repository-root=$LOCATION +``` + +## Customizing container images + +You can add extra dependencies or serialization files to container images so the execution engine doesn't need them. Review comment: This sentence is unclear-- Do you mean that if you add extra dependencies or serialization files, then you don't need to supply them again later? What does it mean that "the execution engine doesn't need them"? What is "them" in this sentence? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317117) Time Spent: 1h 20m (was: 1h 10m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317119 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 24/Sep/19 01:31 Start Date: 24/Sep/19 01:31 Worklog Time Spent: 10m Work Description: rosetn commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327392732 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker Review comment: (I realize this comment might look weird because it gets rendered, but see if you can view the source) I would use code fences instead of the pre ... tags here. I don't see the white/grey background on the Beam site for other code boxes, so maybe change this to remain consistent. Then, you need to rewrite this list in markdown 1. Navigate to your local copy of https://github.com/apache/beam;>beam 1. Run Gradle with the `docker` target: ``` ./gradlew docker ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317119) Time Spent: 1.5h (was: 1h 20m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317112=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317112 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 24/Sep/19 01:31 Start Date: 24/Sep/19 01:31 Worklog Time Spent: 10m Work Description: rosetn commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327393049 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + Review comment: Missing or extra word "Navigate to your local copy of [the]... " This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317112) Time Spent: 1h (was: 50m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317111=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317111 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 24/Sep/19 01:31 Start Date: 24/Sep/19 01:31 Worklog Time Spent: 10m Work Description: rosetn commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327389718 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: Review comment: "may"->"might" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317111) Time Spent: 50m (was: 40m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317115=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317115 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 24/Sep/19 01:31 Start Date: 24/Sep/19 01:31 Worklog Time Spent: 10m Work Description: rosetn commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327393615 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: + +``` +./gradlew docker -Pdocker-tag=2.3.0 +``` + +To change the `docker` target, build the container with the `docker-repository-root` option: + +``` +./gradlew docker -Pdocker-repository-root=$LOCATION +``` + +## Customizing container images + +You can add extra dependencies or serialization files to container images so the execution engine doesn't need them. + +To customize a container image, either: +* [Write a new](#writing-new-dockerfiles) [Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the original +* [Modify](#modifying-dockerfiles) the [original Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile) and reimage the container + +It's often easier to write a new Dockerfile, but you can customize anything, including the base OS, by modifying the original. Review comment: "It's often easier to write a new Dockerfile, but by modifying the original Dockerfile, you can customize anything (including the base OS) ." Just a suggestion--I don't really like parentheses, but this sentence is confusing with too many clauses. Another possibility is to break it up. "It's often easier to write a new Dockerfile. However, by modifying the original Dockerfile, you can customize anything, including the base OS." This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317115) Time Spent: 1h 10m (was: 1h) > Document custom
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317113 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 24/Sep/19 01:31 Start Date: 24/Sep/19 01:31 Worklog Time Spent: 10m Work Description: rosetn commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327390032 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. Review comment: I suggest simplifying "an eponymous repository", maybe "a repository of the same name" ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317113) Time Spent: 1h (was: 50m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317116=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317116
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 24/Sep/19 01:31
Start Date: 24/Sep/19 01:31
Worklog Time Spent: 10m
Work Description: rosetn commented on pull request #9607: [BEAM-8209]
Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r327391188
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
Review comment:
add periods at the end of
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317120=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317120 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 24/Sep/19 01:31 Start Date: 24/Sep/19 01:31 Worklog Time Spent: 10m Work Description: rosetn commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327394115 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: + +``` +REPOSITORY TAGIMAGE IDCREATED SIZE +$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB +$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB +$USER-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB +``` + +Although the respository names look like URLs, the container images are stored locally on your workstation. After building the container images locally, you can [push](#pushing-container-images) them to an eponymous repository online. + +### Overriding default Docker targets + +The default SDK version is `latest` and the default Docker repository is the following Bintray location: + +``` +$USER-docker-apache.bintray.io/beam +``` + +When you [build SDK container images](#building-container-images), you can override the default version and location. + +To specify an older Python SDK version, like 2.3.0, build the container with the `docker-tag` option: + +``` +./gradlew docker -Pdocker-tag=2.3.0 +``` + +To change the `docker` target, build the container with the `docker-repository-root` option: + +``` +./gradlew docker -Pdocker-repository-root=$LOCATION +``` + +## Customizing container images + +You can add extra dependencies or serialization files to container images so the execution engine doesn't need them. + +To customize a container image, either: Review comment: add periods at the end of these sentences This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317120) Time Spent: 1h 40m (was: 1.5h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317109=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317109 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 24/Sep/19 01:31 Start Date: 24/Sep/19 01:31 Worklog Time Spent: 10m Work Description: rosetn commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327388352 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. Review comment: This sentence is confusing and might need to be split. Did you mean: "The Beam SDK runtime environment is isolated from other runtime systems because the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/). This means that any execution engine can run the Beam SDK. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317109) Time Spent: 0.5h (was: 20m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317110=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317110 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 24/Sep/19 01:31 Start Date: 24/Sep/19 01:31 Worklog Time Spent: 10m Work Description: rosetn commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327387167 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: Review comment: I don't understand this redirect--is there more context/rationale? If someone had previously bookmarked the Beam Execution Model page, shouldn't they be redirected to /documentation/runtime/model/ page now? i.e. this redirect should be in the other file This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317110) Time Spent: 40m (was: 0.5h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[
https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317118=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317118
]
ASF GitHub Bot logged work on BEAM-8209:
Author: ASF GitHub Bot
Created on: 24/Sep/19 01:31
Start Date: 24/Sep/19 01:31
Worklog Time Spent: 10m
Work Description: rosetn commented on pull request #9607: [BEAM-8209]
Custom container docs
URL: https://github.com/apache/beam/pull/9607#discussion_r327391740
##
File path: website/src/documentation/runtime/environments.md
##
@@ -0,0 +1,187 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+redirect_from:
+ - /documentation/execution-model/
+---
+
+
+# Runtime environments
+
+Any execution engine can run the Beam SDK beacuse the SDK runtime environment
is [containerized](https://s.apache.org/beam-fn-api-container-contract) with
[Docker](https://www.docker.com/) and isolated from other runtime systems. This
page describes how to build, customize, and push Beam SDK container images.
+
+## Building container images
+
+Before building Beam SDK container images:
+* Register a [Bintray](https://bintray.com/) account with a Docker repository
named `apache`.
+* Install [Docker](https://www.docker.com/) on your workstation.
+
+To build Beam SDK container images:
+
+
+
+Navigate to your local copy of the https://github.com/apache/beam;>beam
+
+
+Run Gradle with the docker target: ./gradlew
docker
+
+
+
+> **Note**: It may take a long time to build all of the container images. You
can instead build the images for specific SDKs:
+>
+> ```
+> ./gradlew -p sdks/java/container docker
+> ./gradlew -p sdks/python/container docker
+> ./gradlew -p sdks/go/container docker
+> ```
+
+Run `docker images` to examine the containers. For example, if you
successfully built the container images, the command prompt displays a response
like:
+
+```
+REPOSITORY TAGIMAGE
IDCREATED SIZE
+$USER-docker-apache.bintray.io/beam/python latest 4ea515403a1a
3 minutes ago 1.27GB
+$USER-docker-apache.bintray.io/beam/java latest 0103512f1d8f
34 minutes ago 780MB
+$USER-docker-apache.bintray.io/beam/go latest ce055985808a
35 minutes ago 121MB
+```
+
+Although the respository names look like URLs, the container images are
stored locally on your workstation. After building the container images
locally, you can [push](#pushing-container-images) them to an eponymous
repository online.
+
+### Overriding default Docker targets
+
+The default SDK version is `latest` and the default Docker repository is the
following Bintray location:
+
+```
+$USER-docker-apache.bintray.io/beam
+```
+
+When you [build SDK container images](#building-container-images), you can
override the default version and location.
+
+To specify an older Python SDK version, like 2.3.0, build the container with
the `docker-tag` option:
+
+```
+./gradlew docker -Pdocker-tag=2.3.0
+```
+
+To change the `docker` target, build the container with the
`docker-repository-root` option:
+
+```
+./gradlew docker -Pdocker-repository-root=$LOCATION
+```
+
+## Customizing container images
+
+You can add extra dependencies or serialization files to container images so
the execution engine doesn't need them.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles)
[Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the
original
+* [Modify](#modifying-dockerfiles) the [original
Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile)
and reimage the container
+
+It's often easier to write a new Dockerfile, but you can customize anything,
including the base OS, by modifying the original.
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+
+
+Pull a https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release;>prebuilt
SDK container image for your target language and version.
+
+
+https://docs.docker.com/develop/develop-images/dockerfile_best-practices/;>Write
a new Dockerfile that https://docs.docker.com/engine/reference/builder/#from;>designates
the original as its https://docs.docker.com/glossary/?term=parent%20image;>parent
+
+
+Build a child image: docker build -f /path/to/new/Dockerfile
+
+
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Pull the [prebuilt SDK container
image](https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release)
for your target language and version
+2. Customize the
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317114=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317114 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 24/Sep/19 01:31 Start Date: 24/Sep/19 01:31 Worklog Time Spent: 10m Work Description: rosetn commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327390222 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. + +## Building container images + +Before building Beam SDK container images: +* Register a [Bintray](https://bintray.com/) account with a Docker repository named `apache`. +* Install [Docker](https://www.docker.com/) on your workstation. + +To build Beam SDK container images: + + + +Navigate to your local copy of the https://github.com/apache/beam;>beam + + +Run Gradle with the docker target: ./gradlew docker + + + +> **Note**: It may take a long time to build all of the container images. You can instead build the images for specific SDKs: +> +> ``` +> ./gradlew -p sdks/java/container docker +> ./gradlew -p sdks/python/container docker +> ./gradlew -p sdks/go/container docker +> ``` + +Run `docker images` to examine the containers. For example, if you successfully built the container images, the command prompt displays a response like: Review comment: "like:"->"like the following:" or "such as the following: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317114) Time Spent: 1h 10m (was: 1h) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317108=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317108 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 24/Sep/19 01:31 Start Date: 24/Sep/19 01:31 Worklog Time Spent: 10m Work Description: rosetn commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#discussion_r327387472 ## File path: website/src/documentation/runtime/environments.md ## @@ -0,0 +1,187 @@ +--- +layout: section +title: "Runtime environments" +section_menu: section-menu/documentation.html +permalink: /documentation/runtime/environments/ +redirect_from: + - /documentation/execution-model/ +--- + + +# Runtime environments + +Any execution engine can run the Beam SDK beacuse the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/) and isolated from other runtime systems. This page describes how to build, customize, and push Beam SDK container images. Review comment: "beacuse"->"because" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317108) Time Spent: 0.5h (was: 20m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=317078=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317078 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 24/Sep/19 00:21 Start Date: 24/Sep/19 00:21 Worklog Time Spent: 10m Work Description: rosetn commented on issue #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607#issuecomment-534335284 STAGED: http://apache-beam-website-pull-requests.storage.googleapis.com/9607/documentation/runtime/environments/index.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317078) Time Spent: 20m (was: 10m) > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?focusedWorklogId=314357=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-314357 ] ASF GitHub Bot logged work on BEAM-8209: Author: ASF GitHub Bot Created on: 18/Sep/19 14:41 Start Date: 18/Sep/19 14:41 Worklog Time Spent: 10m Work Description: soyrice commented on pull request #9607: [BEAM-8209] Custom container docs URL: https://github.com/apache/beam/pull/9607 - Create new section on Beam website about runtime systems - Add how-to guide about customizing SDK container images Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | [