[
https://issues.apache.org/jira/browse/BEAM-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16756282#comment-16756282
]
Mark Liu commented on BEAM-6542:
--------------------------------
If we want to keep two sync, can we generate requirements.txt from setup.py
during build phase? Even we have test to check matches, it still requires
manual update. Probably it can be done by gradle.
> Python ValidatesContainer test suite should verify that installed
> dependencies match dependencies in requirements file.
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-6542
> URL: https://issues.apache.org/jira/browse/BEAM-6542
> Project: Beam
> Issue Type: Improvement
> Components: testing
> Reporter: Valentyn Tymofieiev
> Priority: Minor
>
> Python [ValidatesContainer test
> suites|https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy]
> build Docker containers and run some integration tests using those
> containers, to make sure that containers can be built and used. A python
> container that we build [includes Beam
> SDK|https://github.com/apache/beam/blob/1a6490d3fd9245fc59838bd4bd531755304a855a/sdks/python/container/Dockerfile#L47].
> During container build we install several pip packages, which is influenced
> by
> [requirements.txt|https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt],
> [SDK
> dependencies|https://github.com/apache/beam/blob/d9a1bac19c52b92804a204b7ca881b3e8617b42c/sdks/python/setup.py#L193],
> and downstream dependencies of packages we install.
> The purpose of specifying dependencies in setup.py is to define minimal
> viable requirements for SDK to be installed.
> The purpose of requirements.txt is to configure the runtime environment for
> SDK harness as and requires more precision to make sure there are no version
> dependency conflicts and to make sure that versions of dependencies installed
> in the container match across two different container builds. therefore we
> should specify exact versions in requirements.txt and also include all
> transitive dependencies of Beam.
> Unfortunately, requirements.txt can easily go out of sync even with Beam SDK
> requirements ([example|https://github.com/apache/beam/pull/7657])
> We should strengthen ValidatesContainer test suite to verify that version of
> dependencies installed in the container matches versions specified in
> requirements.txt.
> One possible way to do it is to run `pip freeze` and compare the output with
> requirements.txt.
> cc [~markflyhigh], [~altay].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
