> -----Original Message----- > From: Clark Boylan [mailto:[email protected]] > Sent: 03 September 2014 21:57 > To: [email protected] > Subject: Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt > > > > On Wed, Sep 3, 2014, at 01:06 PM, Kuvaja, Erno wrote: > > > -----Original Message----- > > > From: Clark Boylan [mailto:[email protected]] > > > Sent: 03 September 2014 20:10 > > > To: [email protected] > > > Subject: Re: [openstack-dev] [all] [glance] do NOT ever sort > > > requirements.txt > > > > > > > > > > > > On Wed, Sep 3, 2014, at 11:51 AM, Kuvaja, Erno wrote: > > > > > -----Original Message----- > > > > > From: Sean Dague [mailto:[email protected]] > > > > > Sent: 03 September 2014 13:37 > > > > > To: OpenStack Development Mailing List (not for usage questions) > > > > > Subject: [openstack-dev] [all] [glance] do NOT ever sort > > > > > requirements.txt > > > > > > > > > > I'm not sure why people keep showing up with "sort requirements" > > > > > patches like - https://review.openstack.org/#/c/76817/6, > > > > > however, they > > > do. > > > > > > > > > > All of these need to be -2ed with predjudice. > > > > > > > > > > requirements.txt is not a declarative interface. The order is > > > > > important as pip processes it in the order it is. Changing the > > > > > order has impacts on the overall integration which can cause wedges > later. > > > > > > > > > > So please stop. > > > > > > > > > > -Sean > > > > > > > > > > -- > > > > > Sean Dague > > > > > http://dague.net > > > > > > > > > > > > > Hi Sean & all, > > > > > > > > Could you please open this up a little bit? What are we afraid > > > > breaking regarding the order of these requirements? I tried to go > > > > through pip documentation but I could not find reason of specific > > > > order of the lines, references to keep the order there was 'though. > > > > > > > > I'm now assuming one thing here as I do not know if that's the case. > > > > None of the packages enables/disables functionality depending of > > > > what has been installed on the system before, but they have their > > > > own dependencies to provide those. Based on this assumption I can > > > > think of only one scenario causing us issues. That is us abusing > > > > the example in point 2 of > > > > https://pip.pypa.io/en/latest/user_guide.html#requirements-files > > > > meaning; we install package X depending on package Y>=1.0,<2.0 > > > > before installing package Z depending on Y>=1.0 to ensure that > > > > package Y<2.0 without pinning package Y in our requirements.txt. I > > > > certainly hope that this is not the case as depending 3rd party > > > > vendor providing us specific > > > version of dependency package would be extremely stupid. > > > > > > > > Other than that I really don't know how the order could cause us > > > > issues, but I would be really happy to learn something new today > > > > if that is the case or if my assumption went wrong. > > > > > > > > Best Regards, > > > > Erno (jokke_) Kuvaja > > > > > > > > > _______________________________________________ > > > > > OpenStack-dev mailing list > > > > > [email protected] > > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-de > > > > > v > > > > > > > > _______________________________________________ > > > > OpenStack-dev mailing list > > > > [email protected] > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > The issue is described in the bug that Josh linked > > > (https://github.com/pypa/pip/issues/988). Basically pip doesn't do > > > dependency resolution in a way that lets you treat requirements as > > > order independent. For that to be the case pip would have to > > > evaluate all dependencies together then install the intersection of those > dependencies. > > > Instead it iterates over the list(s) in order and evaluates each > > > dependency as it is found. > > > > > > Your example basically describes where this breaks. You can both > > > depend on the same dependency at different versions and pip will > > > install a version that satisfies only one of the dependencies and > > > not the other leading to a failed install. However I think a more > > > common case is that openstack will pin a dependency and say > > > Y>=1.0,<2.0 and the X dependency will say Y>=1.0. If the X > > > dependency comes first you get version 2.5 which is not valid for your > specification of Y>=1.0,<2.0 and pip fails. > > > You fix this by listing Y before X dependency that installs Y with > > > less restrictive boundaries. > > > > > > Another example of a slightly different failure would be hacking, > > > flake8, pep8, and pyflakes. Hacking installs a specific version of > > > flake8, pep8, and pyflakes so that we do static lint checking with > > > consistent checks each release. If you sort this list alphabetically > > > instead of allowing hacking to install its deps flake8 will come first > > > and you > can get a different version of pep8. > > > Different versions of pep8 check different things and now the gate > > > has broken. > > > > > > The most problematic thing is you can't count on your dependencies > > > from not breaking you if they come first (because they are evaluated > first). > > > So in cases where we know order is important (hacking and pbr and > > > probably a handful of others) we should be listing them as early as > > > possible in the requirements. > > > > > > Clark > > > > > > _______________________________________________ > > > OpenStack-dev mailing list > > > [email protected] > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > Thanks Clark, > > > > To be honest the issue nor your explanation did clarify this to me. > > Please forgive me hunting this, but it seems to be extremely important > > topic so I would like to understand where it's coming from (and > > hopefully others will benefit from it too). > > > > So if I got this right the problem is the dependencies that has not > > been pinned. Even that issue which has been open is pretty clear about > that. > > Also by the pip documentation and the issue text itself if we have > > those dependencies pinned to our limits on the first level (our > > requirements.txt) they will be obeyed regardless where they are on the > > file. Just their dependencies (that are not listed) are not going to > > be cross checked. > > > > Now wouldn't it be right way to attack this problem to actually > > specify the limits so that we have all of them in the requirements > > (and they don't break if the maintainer of that first one happens to > > change the > > dependencies) rather than trusting the order giving us right > > dependencies? > > > Pinning all transitive dependencies in all OpenStack projects so that we can > avoid a handful of conflicts is not worth it. OpenStack has a couple hundred > first level dependencies [0] and it is enough work loosely managing those. > Conflicts arise when two first level dependencies have similar but conflicting > second level or more recursive dependencies that conflict. We can avoid > those conflicts with simple ordering of dependency files which is much easier > than the alternative. > > I do not see any benefit to a naive sort of requirements files but there is a > benefit to giving them a specific order so that we can avoid these conflicts. > > It is also worth nothing that finding new failures as they happen is > beneficial. > It is painful, but without it there is very little in the way of advance > warning. > You want to find this stuff out as soon as possible and not at release time. > > [0] > https://git.openstack.org/cgit/openstack/requirements/tree/global- > requirements.txt > > Clark > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Thanks for taking time to explaining this to me Clark, Suddenly it starts to make sense! - Erno _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
