Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
On 09/03/2014 09:09 PM, Clark Boylan wrote: On Wed, Sep 3, 2014, at 11:51 AM, Kuvaja, Erno wrote: -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 03 September 2014 13:37 To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt I'm not sure why people keep showing up with sort requirements patches like - https://review.openstack.org/#/c/76817/6, however, they do. All of these need to be -2ed with predjudice. requirements.txt is not a declarative interface. The order is important as pip processes it in the order it is. Changing the order has impacts on the overall integration which can cause wedges later. So please stop. -Sean -- Sean Dague http://dague.net Hi Sean all, Could you please open this up a little bit? What are we afraid breaking regarding the order of these requirements? I tried to go through pip documentation but I could not find reason of specific order of the lines, references to keep the order there was 'though. I'm now assuming one thing here as I do not know if that's the case. None of the packages enables/disables functionality depending of what has been installed on the system before, but they have their own dependencies to provide those. Based on this assumption I can think of only one scenario causing us issues. That is us abusing the example in point 2 of https://pip.pypa.io/en/latest/user_guide.html#requirements-files meaning; we install package X depending on package Y=1.0,2.0 before installing package Z depending on Y=1.0 to ensure that package Y2.0 without pinning package Y in our requirements.txt. I certainly hope that this is not the case as depending 3rd party vendor providing us specific version of dependency package would be extremely stupid. Other than that I really don't know how the order could cause us issues, but I would be really happy to learn something new today if that is the case or if my assumption went wrong. Best Regards, Erno (jokke_) Kuvaja ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev The issue is described in the bug that Josh linked (https://github.com/pypa/pip/issues/988). Basically pip doesn't do dependency resolution in a way that lets you treat requirements as order independent. For that to be the case pip would have to evaluate all dependencies together then install the intersection of those dependencies. Instead it iterates over the list(s) in order and evaluates each dependency as it is found. Your example basically describes where this breaks. You can both depend on the same dependency at different versions and pip will install a version that satisfies only one of the dependencies and not the other leading to a failed install. However I think a more common case is that openstack will pin a dependency and say Y=1.0,2.0 and the X dependency will say Y=1.0. If the X dependency comes first you get version 2.5 which is not valid for your specification of Y=1.0,2.0 and pip fails. You fix this by listing Y before X dependency that installs Y with less restrictive boundaries. Another example of a slightly different failure would be hacking, flake8, pep8, and pyflakes. Hacking installs a specific version of flake8, pep8, and pyflakes so that we do static lint checking with consistent checks each release. If you sort this list alphabetically instead of allowing hacking to install its deps flake8 will come first and you can get a different version of pep8. Different versions of pep8 check different things and now the gate has broken. The most problematic thing is you can't count on your dependencies from not breaking you if they come first (because they are evaluated first). So in cases where we know order is important (hacking and pbr and probably a handful of others) we should be listing them as early as possible in the requirements. So, is there a specific order to look out for? AFAIU requirements should have pbr as first requirement and test-requirements should have hacking as first one. Is there anything else? What's the best place to document this? Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
On Wed, Sep 03, 2014 at 08:37:17AM -0400, Sean Dague wrote: I'm not sure why people keep showing up with sort requirements patches like - https://review.openstack.org/#/c/76817/6, however, they do. All of these need to be -2ed with predjudice. requirements.txt is not a declarative interface. The order is important as pip processes it in the order it is. Changing the order has impacts on the overall integration which can cause wedges later. Can requirements.txt contain comment lines ? If so, it would be worth adding # The ordering of modules in this file is important # Do not attempt to re-sort the lines Because 6 months hence people will have probably forgotten about this mail, or if they're new contributors, never know it existed. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
On Wed, 2014-09-03 at 14:03 +0100, Daniel P. Berrange wrote: On Wed, Sep 03, 2014 at 08:37:17AM -0400, Sean Dague wrote: I'm not sure why people keep showing up with sort requirements patches like - https://review.openstack.org/#/c/76817/6, however, they do. All of these need to be -2ed with predjudice. requirements.txt is not a declarative interface. The order is important as pip processes it in the order it is. Changing the order has impacts on the overall integration which can cause wedges later. Can requirements.txt contain comment lines ? If so, it would be worth adding # The ordering of modules in this file is important # Do not attempt to re-sort the lines Because 6 months hence people will have probably forgotten about this mail, or if they're new contributors, never know it existed. Regards, Daniel Yes, requirements.txt can contain comment lines. It's a good idea to keep this information in the file itself. Best, Romain ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
On 09/03/2014 09:03 AM, Daniel P. Berrange wrote: On Wed, Sep 03, 2014 at 08:37:17AM -0400, Sean Dague wrote: I'm not sure why people keep showing up with sort requirements patches like - https://review.openstack.org/#/c/76817/6, however, they do. All of these need to be -2ed with predjudice. requirements.txt is not a declarative interface. The order is important as pip processes it in the order it is. Changing the order has impacts on the overall integration which can cause wedges later. Can requirements.txt contain comment lines ? If so, it would be worth adding # The ordering of modules in this file is important # Do not attempt to re-sort the lines Because 6 months hence people will have probably forgotten about this mail, or if they're new contributors, never know it existed. The point is that core review team members should know. In this case at least one glance core +2ed this change. Regular contributors can be educated by core team members. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
On Wed, Sep 3, 2014 at 8:25 AM, Sean Dague s...@dague.net wrote: On 09/03/2014 09:03 AM, Daniel P. Berrange wrote: On Wed, Sep 03, 2014 at 08:37:17AM -0400, Sean Dague wrote: I'm not sure why people keep showing up with sort requirements patches like - https://review.openstack.org/#/c/76817/6, however, they do. All of these need to be -2ed with predjudice. requirements.txt is not a declarative interface. The order is important as pip processes it in the order it is. Changing the order has impacts on the overall integration which can cause wedges later. Can requirements.txt contain comment lines ? If so, it would be worth adding # The ordering of modules in this file is important # Do not attempt to re-sort the lines Because 6 months hence people will have probably forgotten about this mail, or if they're new contributors, never know it existed. The point is that core review team members should know. In this case at least one glance core +2ed this change. Regular contributors can be educated by core team members. Regardless, tribal knowledge should be documented, and doing so in requirements files is probably the best place for that. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
On Sep 3, 2014, at 12:20 PM, Dolph Mathews dolph.math...@gmail.com wrote: On Wed, Sep 3, 2014 at 8:25 AM, Sean Dague s...@dague.net wrote: On 09/03/2014 09:03 AM, Daniel P. Berrange wrote: On Wed, Sep 03, 2014 at 08:37:17AM -0400, Sean Dague wrote: I'm not sure why people keep showing up with sort requirements patches like - https://review.openstack.org/#/c/76817/6, however, they do. All of these need to be -2ed with predjudice. requirements.txt is not a declarative interface. The order is important as pip processes it in the order it is. Changing the order has impacts on the overall integration which can cause wedges later. Can requirements.txt contain comment lines ? If so, it would be worth adding # The ordering of modules in this file is important # Do not attempt to re-sort the lines Because 6 months hence people will have probably forgotten about this mail, or if they're new contributors, never know it existed. The point is that core review team members should know. In this case at least one glance core +2ed this change. Regular contributors can be educated by core team members. Regardless, tribal knowledge should be documented, and doing so in requirements files is probably the best place for that. +1 Write-it-down-ly, Doug -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
On Wed, Sep 3, 2014 at 11:23 AM, Doug Hellmann d...@doughellmann.com wrote: On Sep 3, 2014, at 12:20 PM, Dolph Mathews dolph.math...@gmail.com wrote: On Wed, Sep 3, 2014 at 8:25 AM, Sean Dague s...@dague.net wrote: On 09/03/2014 09:03 AM, Daniel P. Berrange wrote: On Wed, Sep 03, 2014 at 08:37:17AM -0400, Sean Dague wrote: I'm not sure why people keep showing up with sort requirements patches like - https://review.openstack.org/#/c/76817/6, however, they do. All of these need to be -2ed with predjudice. requirements.txt is not a declarative interface. The order is important as pip processes it in the order it is. Changing the order has impacts on the overall integration which can cause wedges later. Can requirements.txt contain comment lines ? If so, it would be worth adding # The ordering of modules in this file is important # Do not attempt to re-sort the lines Because 6 months hence people will have probably forgotten about this mail, or if they're new contributors, never know it existed. The point is that core review team members should know. In this case at least one glance core +2ed this change. Regular contributors can be educated by core team members. Regardless, tribal knowledge should be documented, and doing so in requirements files is probably the best place for that. +1 Write-it-down-ly, Doug The blocked review Sean mentioned above happens to reference a bug that I've now marked invalid against all projects that hadn't already applied fixes for it: https://bugs.launchpad.net/glance/+bug/1285478 Instead, I've opened a second task to add a note to all the requirements files: https://bugs.launchpad.net/keystone/+bug/1365061 All associated patches: https://review.openstack.org/#/q/I64ae9191863564e278a35d42ec9cd743a233028e,n,z -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
-Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 03 September 2014 13:37 To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt I'm not sure why people keep showing up with sort requirements patches like - https://review.openstack.org/#/c/76817/6, however, they do. All of these need to be -2ed with predjudice. requirements.txt is not a declarative interface. The order is important as pip processes it in the order it is. Changing the order has impacts on the overall integration which can cause wedges later. So please stop. -Sean -- Sean Dague http://dague.net Hi Sean all, Could you please open this up a little bit? What are we afraid breaking regarding the order of these requirements? I tried to go through pip documentation but I could not find reason of specific order of the lines, references to keep the order there was 'though. I'm now assuming one thing here as I do not know if that's the case. None of the packages enables/disables functionality depending of what has been installed on the system before, but they have their own dependencies to provide those. Based on this assumption I can think of only one scenario causing us issues. That is us abusing the example in point 2 of https://pip.pypa.io/en/latest/user_guide.html#requirements-files meaning; we install package X depending on package Y=1.0,2.0 before installing package Z depending on Y=1.0 to ensure that package Y2.0 without pinning package Y in our requirements.txt. I certainly hope that this is not the case as depending 3rd party vendor providing us specific version of dependency package would be extremely stupid. Other than that I really don't know how the order could cause us issues, but I would be really happy to learn something new today if that is the case or if my assumption went wrong. Best Regards, Erno (jokke_) Kuvaja ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
On Wed, Sep 3, 2014, at 11:51 AM, Kuvaja, Erno wrote: -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 03 September 2014 13:37 To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt I'm not sure why people keep showing up with sort requirements patches like - https://review.openstack.org/#/c/76817/6, however, they do. All of these need to be -2ed with predjudice. requirements.txt is not a declarative interface. The order is important as pip processes it in the order it is. Changing the order has impacts on the overall integration which can cause wedges later. So please stop. -Sean -- Sean Dague http://dague.net Hi Sean all, Could you please open this up a little bit? What are we afraid breaking regarding the order of these requirements? I tried to go through pip documentation but I could not find reason of specific order of the lines, references to keep the order there was 'though. I'm now assuming one thing here as I do not know if that's the case. None of the packages enables/disables functionality depending of what has been installed on the system before, but they have their own dependencies to provide those. Based on this assumption I can think of only one scenario causing us issues. That is us abusing the example in point 2 of https://pip.pypa.io/en/latest/user_guide.html#requirements-files meaning; we install package X depending on package Y=1.0,2.0 before installing package Z depending on Y=1.0 to ensure that package Y2.0 without pinning package Y in our requirements.txt. I certainly hope that this is not the case as depending 3rd party vendor providing us specific version of dependency package would be extremely stupid. Other than that I really don't know how the order could cause us issues, but I would be really happy to learn something new today if that is the case or if my assumption went wrong. Best Regards, Erno (jokke_) Kuvaja ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev The issue is described in the bug that Josh linked (https://github.com/pypa/pip/issues/988). Basically pip doesn't do dependency resolution in a way that lets you treat requirements as order independent. For that to be the case pip would have to evaluate all dependencies together then install the intersection of those dependencies. Instead it iterates over the list(s) in order and evaluates each dependency as it is found. Your example basically describes where this breaks. You can both depend on the same dependency at different versions and pip will install a version that satisfies only one of the dependencies and not the other leading to a failed install. However I think a more common case is that openstack will pin a dependency and say Y=1.0,2.0 and the X dependency will say Y=1.0. If the X dependency comes first you get version 2.5 which is not valid for your specification of Y=1.0,2.0 and pip fails. You fix this by listing Y before X dependency that installs Y with less restrictive boundaries. Another example of a slightly different failure would be hacking, flake8, pep8, and pyflakes. Hacking installs a specific version of flake8, pep8, and pyflakes so that we do static lint checking with consistent checks each release. If you sort this list alphabetically instead of allowing hacking to install its deps flake8 will come first and you can get a different version of pep8. Different versions of pep8 check different things and now the gate has broken. The most problematic thing is you can't count on your dependencies from not breaking you if they come first (because they are evaluated first). So in cases where we know order is important (hacking and pbr and probably a handful of others) we should be listing them as early as possible in the requirements. Clark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
-Original Message- From: Clark Boylan [mailto:cboy...@sapwetik.org] Sent: 03 September 2014 20:10 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt On Wed, Sep 3, 2014, at 11:51 AM, Kuvaja, Erno wrote: -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 03 September 2014 13:37 To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt I'm not sure why people keep showing up with sort requirements patches like - https://review.openstack.org/#/c/76817/6, however, they do. All of these need to be -2ed with predjudice. requirements.txt is not a declarative interface. The order is important as pip processes it in the order it is. Changing the order has impacts on the overall integration which can cause wedges later. So please stop. -Sean -- Sean Dague http://dague.net Hi Sean all, Could you please open this up a little bit? What are we afraid breaking regarding the order of these requirements? I tried to go through pip documentation but I could not find reason of specific order of the lines, references to keep the order there was 'though. I'm now assuming one thing here as I do not know if that's the case. None of the packages enables/disables functionality depending of what has been installed on the system before, but they have their own dependencies to provide those. Based on this assumption I can think of only one scenario causing us issues. That is us abusing the example in point 2 of https://pip.pypa.io/en/latest/user_guide.html#requirements-files meaning; we install package X depending on package Y=1.0,2.0 before installing package Z depending on Y=1.0 to ensure that package Y2.0 without pinning package Y in our requirements.txt. I certainly hope that this is not the case as depending 3rd party vendor providing us specific version of dependency package would be extremely stupid. Other than that I really don't know how the order could cause us issues, but I would be really happy to learn something new today if that is the case or if my assumption went wrong. Best Regards, Erno (jokke_) Kuvaja ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev The issue is described in the bug that Josh linked (https://github.com/pypa/pip/issues/988). Basically pip doesn't do dependency resolution in a way that lets you treat requirements as order independent. For that to be the case pip would have to evaluate all dependencies together then install the intersection of those dependencies. Instead it iterates over the list(s) in order and evaluates each dependency as it is found. Your example basically describes where this breaks. You can both depend on the same dependency at different versions and pip will install a version that satisfies only one of the dependencies and not the other leading to a failed install. However I think a more common case is that openstack will pin a dependency and say Y=1.0,2.0 and the X dependency will say Y=1.0. If the X dependency comes first you get version 2.5 which is not valid for your specification of Y=1.0,2.0 and pip fails. You fix this by listing Y before X dependency that installs Y with less restrictive boundaries. Another example of a slightly different failure would be hacking, flake8, pep8, and pyflakes. Hacking installs a specific version of flake8, pep8, and pyflakes so that we do static lint checking with consistent checks each release. If you sort this list alphabetically instead of allowing hacking to install its deps flake8 will come first and you can get a different version of pep8. Different versions of pep8 check different things and now the gate has broken. The most problematic thing is you can't count on your dependencies from not breaking you if they come first (because they are evaluated first). So in cases where we know order is important (hacking and pbr and probably a handful of others) we should be listing them as early as possible in the requirements. Clark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Thanks Clark, To be honest the issue nor your explanation did clarify this to me. Please forgive me hunting this, but it seems to be extremely important topic so I would like to understand where it's coming from (and hopefully others will benefit from
Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
On Wed, Sep 3, 2014, at 01:06 PM, Kuvaja, Erno wrote: -Original Message- From: Clark Boylan [mailto:cboy...@sapwetik.org] Sent: 03 September 2014 20:10 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt On Wed, Sep 3, 2014, at 11:51 AM, Kuvaja, Erno wrote: -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 03 September 2014 13:37 To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt I'm not sure why people keep showing up with sort requirements patches like - https://review.openstack.org/#/c/76817/6, however, they do. All of these need to be -2ed with predjudice. requirements.txt is not a declarative interface. The order is important as pip processes it in the order it is. Changing the order has impacts on the overall integration which can cause wedges later. So please stop. -Sean -- Sean Dague http://dague.net Hi Sean all, Could you please open this up a little bit? What are we afraid breaking regarding the order of these requirements? I tried to go through pip documentation but I could not find reason of specific order of the lines, references to keep the order there was 'though. I'm now assuming one thing here as I do not know if that's the case. None of the packages enables/disables functionality depending of what has been installed on the system before, but they have their own dependencies to provide those. Based on this assumption I can think of only one scenario causing us issues. That is us abusing the example in point 2 of https://pip.pypa.io/en/latest/user_guide.html#requirements-files meaning; we install package X depending on package Y=1.0,2.0 before installing package Z depending on Y=1.0 to ensure that package Y2.0 without pinning package Y in our requirements.txt. I certainly hope that this is not the case as depending 3rd party vendor providing us specific version of dependency package would be extremely stupid. Other than that I really don't know how the order could cause us issues, but I would be really happy to learn something new today if that is the case or if my assumption went wrong. Best Regards, Erno (jokke_) Kuvaja ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev The issue is described in the bug that Josh linked (https://github.com/pypa/pip/issues/988). Basically pip doesn't do dependency resolution in a way that lets you treat requirements as order independent. For that to be the case pip would have to evaluate all dependencies together then install the intersection of those dependencies. Instead it iterates over the list(s) in order and evaluates each dependency as it is found. Your example basically describes where this breaks. You can both depend on the same dependency at different versions and pip will install a version that satisfies only one of the dependencies and not the other leading to a failed install. However I think a more common case is that openstack will pin a dependency and say Y=1.0,2.0 and the X dependency will say Y=1.0. If the X dependency comes first you get version 2.5 which is not valid for your specification of Y=1.0,2.0 and pip fails. You fix this by listing Y before X dependency that installs Y with less restrictive boundaries. Another example of a slightly different failure would be hacking, flake8, pep8, and pyflakes. Hacking installs a specific version of flake8, pep8, and pyflakes so that we do static lint checking with consistent checks each release. If you sort this list alphabetically instead of allowing hacking to install its deps flake8 will come first and you can get a different version of pep8. Different versions of pep8 check different things and now the gate has broken. The most problematic thing is you can't count on your dependencies from not breaking you if they come first (because they are evaluated first). So in cases where we know order is important (hacking and pbr and probably a handful of others) we should be listing them as early as possible in the requirements. Clark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Thanks Clark, To be honest the issue nor your explanation did clarify
Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
-Original Message- From: Clark Boylan [mailto:cboy...@sapwetik.org] Sent: 03 September 2014 21:57 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt On Wed, Sep 3, 2014, at 01:06 PM, Kuvaja, Erno wrote: -Original Message- From: Clark Boylan [mailto:cboy...@sapwetik.org] Sent: 03 September 2014 20:10 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt On Wed, Sep 3, 2014, at 11:51 AM, Kuvaja, Erno wrote: -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 03 September 2014 13:37 To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt I'm not sure why people keep showing up with sort requirements patches like - https://review.openstack.org/#/c/76817/6, however, they do. All of these need to be -2ed with predjudice. requirements.txt is not a declarative interface. The order is important as pip processes it in the order it is. Changing the order has impacts on the overall integration which can cause wedges later. So please stop. -Sean -- Sean Dague http://dague.net Hi Sean all, Could you please open this up a little bit? What are we afraid breaking regarding the order of these requirements? I tried to go through pip documentation but I could not find reason of specific order of the lines, references to keep the order there was 'though. I'm now assuming one thing here as I do not know if that's the case. None of the packages enables/disables functionality depending of what has been installed on the system before, but they have their own dependencies to provide those. Based on this assumption I can think of only one scenario causing us issues. That is us abusing the example in point 2 of https://pip.pypa.io/en/latest/user_guide.html#requirements-files meaning; we install package X depending on package Y=1.0,2.0 before installing package Z depending on Y=1.0 to ensure that package Y2.0 without pinning package Y in our requirements.txt. I certainly hope that this is not the case as depending 3rd party vendor providing us specific version of dependency package would be extremely stupid. Other than that I really don't know how the order could cause us issues, but I would be really happy to learn something new today if that is the case or if my assumption went wrong. Best Regards, Erno (jokke_) Kuvaja ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-de v ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev The issue is described in the bug that Josh linked (https://github.com/pypa/pip/issues/988). Basically pip doesn't do dependency resolution in a way that lets you treat requirements as order independent. For that to be the case pip would have to evaluate all dependencies together then install the intersection of those dependencies. Instead it iterates over the list(s) in order and evaluates each dependency as it is found. Your example basically describes where this breaks. You can both depend on the same dependency at different versions and pip will install a version that satisfies only one of the dependencies and not the other leading to a failed install. However I think a more common case is that openstack will pin a dependency and say Y=1.0,2.0 and the X dependency will say Y=1.0. If the X dependency comes first you get version 2.5 which is not valid for your specification of Y=1.0,2.0 and pip fails. You fix this by listing Y before X dependency that installs Y with less restrictive boundaries. Another example of a slightly different failure would be hacking, flake8, pep8, and pyflakes. Hacking installs a specific version of flake8, pep8, and pyflakes so that we do static lint checking with consistent checks each release. If you sort this list alphabetically instead of allowing hacking to install its deps flake8 will come first and you can get a different version of pep8. Different versions of pep8 check different things and now the gate has broken. The most problematic thing is you can't count on your dependencies from not breaking you if they come first (because they are evaluated first). So in cases where we know order is important (hacking and pbr and probably a handful of others) we should