[
https://issues.apache.org/jira/browse/BEAM-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16865975#comment-16865975
]
Valentyn Tymofieiev commented on BEAM-7474:
-------------------------------------------
[~frederik], you can test your changes by building a container image, for
example, ./gradlew :sdks:python:container:py3:docker, and running
ValidatesContainer suites, for example:
https://github.com/apache/beam/blob/7e633f34e3a8ee68274d2752382fb6a52b5e9b4a/.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy#L25.
The latter suite currently does not pass due to BEAM-7063, but you could still
try to exercise the container with Portable Python Precommit suite, see
https://github.com/apache/beam/pull/8745/files#diff-08a37e2757c9df3f0e1d0ba3e2121eecR101.
Portable Python Precommit suite is currently flaky
(https://issues.apache.org/jira/browse/BEAM-7546), so I don't think we should
run it for all of 3.5 - 3.7, but we could switch Portable Python Precommit from
Python 3.5 to 3.7 after this issue is completed and/or to help test the changes
being made here.
> Add SDK harness containers for Py 3.6, Py 3.7
> ---------------------------------------------
>
> Key: BEAM-7474
> URL: https://issues.apache.org/jira/browse/BEAM-7474
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
> Reporter: Valentyn Tymofieiev
> Assignee: Frederik Bode
> Priority: Major
>
> Currently we can build a Py3-compatible container image with gradle by
> running:
> ./gradlew :sdks:python:container:py3:docker
> This builds a docker container image like:
> valentyn-docker-apache.bintray.io/beam/python3
> The code for this is defined in:
> https://github.com/apache/beam/blob/ae60a72b03f3a2b6b2a06667ec1868a7acc8e38f/sdks/python/container/py3/build.gradle#L48
> To support portable runners that use a container (e.g. Flink) on multiple
> versions of Python 3, we should make it possible to build Python
> 3-compatible SDK harness containers bundled with any desired python version.
> We could have several gradle projects:
> :sdks:python:container:py35:docker
> :sdks:python:container:py36:docker
> :sdks:python:container:py37:docker
> and several Dockerfiles to support this:
>
> sdks/python/container/py35/Dockerfile
> sdks/python/container/py36/Dockerfile
> sdks/python/container/py37/Dockerfile
> The only difference right now would be the base image used in FROM field in
> Dockerfile.
> Alternatively, we could have one parameterized Dockerfile that starts with :
> {code}
> ARG BASE_IMAGE
> FROM $BASE_IMAGE
> ...
> {code}
> I think the latter approach, may result in complications later if these
> containers will need to diverge down the road.
> cc'ing a few folks who may have some feedback on this: [~angoenka] [~mxm]
> [~robertwb] [~Juta] [~frederik].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
