One thing to point out here. Right now if you `pip install apache-airflow=1.10.0` in a clean environment it will fail.
This is because we pin flask-login to 0.2.1 but flask-appbuilder is >= 1.11.1, so that pulls in 1.12.0 which requires flask-login >= 0.3. So I do think there is maybe something to be said about pinning for releases. The down side to that is that if there are updates to a module that we want then we have to make a point release to let people get it Both methods have draw-backs -ash > On 4 Oct 2018, at 17:13, Arthur Wiedmer <arthur.wied...@gmail.com> wrote: > > Hi Jarek, > > I will +1 the discussion Dan is referring to and George's advice. > > I just want to double check we are talking about pinning in > requirements.txt only. > > This offers the ability to > pip install -r requirements.txt > pip install --no-deps airflow > For a guaranteed install which works. > > Several different requirement files can be provided for specific use cases, > like a stable dev one for instance for people wanting to work on operators > and non-core functions. > > However, I think we should proactively test in CI against unpinned > dependencies (though it might be a separate case in the matrix) , so that > we get advance warning if possible that things will break. > CI downtime is not a bad thing here, it actually caught a problem :) > > We should unpin as possible in setup.py to only maintain minimum required > compatibility. The process of pinning in setup.py is extremely detrimental > when you have a large number of python libraries installed with different > pinned versions. > > Best, > Arthur > > On Thu, Oct 4, 2018 at 8:36 AM Dan Davydov <ddavy...@twitter.com.invalid> > wrote: > >> Relevant discussion about this: >> >> https://github.com/apache/incubator-airflow/pull/1809#issuecomment-257502174 >> >> On Thu, Oct 4, 2018 at 11:25 AM Jarek Potiuk <jarek.pot...@polidea.com> >> wrote: >> >>> TL;DR; A change is coming in the way how dependencies/requirements are >>> specified for Apache Airflow - they will be fixed rather than flexible >> (== >>> rather than >=). >>> >>> This is follow up after Slack discussion we had with Ash and Kaxil - >>> summarising what we propose we'll do. >>> >>> *Problem:* >>> During last few weeks we experienced quite a few downtimes of TravisCI >>> builds (for all PRs/branches including master) as some of the transitive >>> dependencies were automatically upgraded. This because in a number of >>> dependencies we have >= rather than == dependencies. >>> >>> Whenever there is a new release of such dependency, it might cause chain >>> reaction with upgrade of transitive dependencies which might get into >>> conflict. >>> >>> An example was Flask-AppBuilder vs flask-login transitive dependency with >>> click. They started to conflict once AppBuilder has released version >>> 1.12.0. >>> >>> *Diagnosis:* >>> Transitive dependencies with "flexible" versions (where >= is used >> instead >>> of ==) is a reason for "dependency hell". We will sooner or later hit >> other >>> cases where not fixed dependencies cause similar problems with other >>> transitive dependencies. We need to fix-pin them. This causes problems >> for >>> both - released versions (cause they stop to work!) and for development >>> (cause they break master builds in TravisCI and prevent people from >>> installing development environment from the scratch. >>> >>> *Solution:* >>> >>> - Following the old-but-good post >>> https://nvie.com/posts/pin-your-packages/ we are going to fix the >>> pinned >>> dependencies to specific versions (so basically all dependencies are >>> "fixed"). >>> - We will introduce mechanism to be able to upgrade dependencies with >>> pip-tools (https://github.com/jazzband/pip-tools). We might also >> take a >>> look at pipenv: https://pipenv.readthedocs.io/en/latest/ >>> - People who would like to upgrade some dependencies for their PRs >> will >>> still be able to do it - but such upgrades will be in their PR thus >> they >>> will go through TravisCI tests and they will also have to be specified >>> with >>> pinned fixed versions (==). This should be part of review process to >>> make >>> sure new/changed requirements are pinned. >>> - In release process there will be a point where an upgrade will be >>> attempted for all requirements (using pip-tools) so that we are not >>> stuck >>> with older releases. This will be in controlled PR environment where >>> there >>> will be time to fix all dependencies without impacting others and >> likely >>> enough time to "vet" such changes (this can be done for alpha/beta >>> releases >>> for example). >>> - As a side effect dependencies specification will become far simpler >>> and straightforward. >>> >>> Happy to hear community comments to the proposal. I am happy to take a >> lead >>> on that, open JIRA issue and implement if this is something community is >>> happy with. >>> >>> J. >>> >>> -- >>> >>> *Jarek Potiuk, Principal Software Engineer* >>> Mobile: +48 660 796 129 >>> >>
