[
https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16223716#comment-16223716
]
Maximilian Roos edited comment on BEAM-3106 at 10/28/17 7:40 PM:
-----------------------------------------------------------------
??What do you think about a policy like, reviewing capped dependencies at every
release and ensuring that a) we are including latest versions of these known
dependencies, b) we are testing with those dependencies before a release.
??
I think that's a reasonable compromise, thanks Ahmet.
To close this off, here's the system that works really well in the numerical
python ecosystem (e.g. pandas / numpy / xarray):
- Run CI tests on a number of released versions of each dependency
- Any backward incompatible changes in dependencies are deprecated in advance,
and tests catch those deprecation warnings - giving plenty of time for changes
(this relies on all dependencies raising warnings for backward-incompatible
changes)
- In the extensively maintained libraries, tests are also run on master branch
of dependencies, to quickly flag any potential breakages
was (Author: maximilianr):
??
What do you think about a policy like, reviewing capped dependencies at every
release and ensuring that a) we are including latest versions of these known
dependencies, b) we are testing with those dependencies before a release.
??
I think that's a reasonable compromise, thanks Ahmet.
To close this off, here's the system that works really well in the numerical
python ecosystem (e.g. pandas / numpy / xarray):
- Run CI tests on a number of released versions of each dependency
- Any backward incompatible changes in dependencies are deprecated in advance,
and tests catch those deprecation warnings - giving plenty of time for changes
(this relies on all dependencies raising warnings for backward-incompatible
changes)
- In the extensively maintained libraries, tests are also run on master branch
of dependencies, to quickly flag any potential breakages
> Consider not pinning all python dependencies, or moving them to
> requirements.txt
> --------------------------------------------------------------------------------
>
> Key: BEAM-3106
> URL: https://issues.apache.org/jira/browse/BEAM-3106
> Project: Beam
> Issue Type: Wish
> Components: build-system
> Affects Versions: 2.1.0
> Environment: python
> Reporter: Maximilian Roos
> Assignee: Ahmet Altay
>
> Currently all python dependencies are [pinned or
> capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97]
> While there's a good argument for supplying a `requirements.txt` with well
> tested dependencies, having them specified in `setup.py` forces them to an
> exact state on each install of Beam. This makes using Beam in any environment
> with other libraries nigh on impossible.
> This is particularly severe for the `gcp` dependencies, where we have
> libraries that won't work with an older version (but Beam _does_ work with an
> newer version). We have to do a bunch of gymnastics to get the correct
> versions installed because of this. Unfortunately, airflow repeats this
> practice and conflicts on a number of dependencies, adding further
> complication (but, again there is no real conflict).
> I haven't seen this practice outside of the Apache & Google ecosystem - for
> example no libraries in numerical python do this. Here's a [discussion on
> SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
