[
https://issues.apache.org/jira/browse/BEAM-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755463#comment-16755463
]
Valentyn Tymofieiev commented on BEAM-6542:
-------------------------------------------
We should also catch errors like: "google-api-core 1.7.0 has requirement
futures>=3.2.0; python_version < "3.2", but you'll have futures 3.1.1 which is
incompatible." during Beam SDK installation.
> Python ValidatesContainer test suite should verify that installed
> dependencies match dependencies in requirements file.
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-6542
> URL: https://issues.apache.org/jira/browse/BEAM-6542
> Project: Beam
> Issue Type: Improvement
> Components: testing
> Reporter: Valentyn Tymofieiev
> Priority: Minor
>
> Python [ValidatesContainer test
> suites|https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy]
> build Docker containers and run some integration tests using those
> containers, to make sure that containers can be built and used. A python
> container that we build [includes Beam
> SDK|https://github.com/apache/beam/blob/1a6490d3fd9245fc59838bd4bd531755304a855a/sdks/python/container/Dockerfile#L47].
> During container build we install several pip packages, which is influenced
> by
> [requirements.txt|https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt],
> [SDK
> dependencies|https://github.com/apache/beam/blob/d9a1bac19c52b92804a204b7ca881b3e8617b42c/sdks/python/setup.py#L193],
> and downstream dependencies of packages we install.
> The purpose of specifying dependencies in setup.py is to define minimal
> viable requirements for SDK to be installed.
> The purpose of requirements.txt is to configure the runtime environment for
> SDK harness as and requires more precision to make sure there are no version
> dependency conflicts and to make sure that versions of dependencies installed
> in the container match across two different container builds. therefore we
> should specify exact versions in requirements.txt and also include all
> transitive dependencies of Beam.
> Unfortunately, requirements.txt can easily go out of sync even with Beam SDK
> requirements ([example|https://github.com/apache/beam/pull/7657])
> We should strengthen ValidatesContainer test suite to verify that version of
> dependencies installed in the container matches versions specified in
> requirements.txt.
> One possible way to do it is to run `pip freeze` and compare the output with
> requirements.txt.
> cc [~markflyhigh], [~altay].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
