Valentyn Tymofieiev created BEAM-6542:
-----------------------------------------
Summary: Python ValidatesContainer test suite should verify that
installed dependencies match dependencies in requirements file.
Key: BEAM-6542
URL: https://issues.apache.org/jira/browse/BEAM-6542
Project: Beam
Issue Type: Improvement
Components: sdk-py-core
Reporter: Valentyn Tymofieiev
Assignee: Ahmet Altay
Python [ValidatesContainer test
suites|https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy]
build Docker containers and run some integration tests using those containers,
to make sure that containers can be built and used. A python container that we
build [includes Beam
SDK|https://github.com/apache/beam/blob/1a6490d3fd9245fc59838bd4bd531755304a855a/sdks/python/container/Dockerfile#L47].
During container build we install several pip packages, which is influenced by
[requirements.txt|https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt],
[SDK
dependencies|https://github.com/apache/beam/blob/d9a1bac19c52b92804a204b7ca881b3e8617b42c/sdks/python/setup.py#L193],
and downstream dependencies of packages we install.
The purpose of specifying dependencies in setup.py is to define minimal viable
requirements for SDK to be installed.
The purpose of requirements.txt is to configure the runtime environment for SDK
harness as and requires more precision to make sure there are no version
dependency conflicts and to make sure that versions of dependencies installed
in the container match across two different container builds. therefore we
should specify exact versions in requirements.txt and also include all
transitive dependencies of Beam.
Unfortunately, requirements.txt can easily go out of sync even with Beam SDK
requirements ([example|https://github.com/apache/beam/pull/7657])
We should strengthen ValidatesContainer test suite to verify that version of
dependencies installed in the container matches versions specified in
requirements.txt.
One possible way to do it is to run `pip freeze` and compare the output with
requirements.txt.
cc [~markflyhigh], [~altay].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
