Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
For those that have been following this escapade I wrote up the
following, hopefully it's useful to someone ;)
https://github.com/harlowja/pippin#how-it-works
Example(s) of actual runs are also @
https://github.com/harlowja/pippin/tree/master/examples
^ Does not include the full output files (since it's a lot of
information/data that wouldn't fit easily on git...)
-Josh
Joshua Harlow wrote:
Cool,
I've got to try that out today to see what it's doing.
I've also shoved my little program up @
https://github.com/harlowja/pippin (the pip-tools one is definitely more
elegantly coded than mine, haha).
Feel free to fork it (modify, run, or ...)
Basic instructions to use it:
https://github.com/harlowja/pippin#pippin
-Josh
Bailey, Darragh wrote:
You may find the code for pip-compile
https://github.com/nvie/pip-tools/tree/future of interest for this, as I
think they may already have a solution for the deep dependency analysis.
I've started experimenting with it for git-upstream cause GitPython have
a habbit of breaking stuff through a couple of releases now :-(
What I like is:
* Doesn't require an extra tool before using 'pip install'
** Some may want to regen the dependencies, but it's optional and the
common python dev approach is retained
* Stable releases are guaranteed to use the versions of dependencies
they were released and verified against
* Improves on the guarantee of gated branch CI
** The idea that if you sync with upstream any test failures are due to
your local changes
** Which is not always true if updated deps can break stuff
On the flip side:
* You remain exposed to security issues in python code until you
manually update
* Development cycle doesn't move forward automatically, may not see
compatibility issues until late when forced to move forward one of the
deps
Think the cons can be handled by some additional CI jobs to update the
pins on a regular basis and pass it through the standard gates and
potentially to auto approve during development cycles if they pass
(already getting the latest matching ones so no big diff here). Some
decisions on trade off around whether this should be done for stable
releases automatically or periodically requiring manual approval would
have to be made.
Did I say how much I like the fact that it doesn't require another tool
before just being able to use 'pip install'?
To experiment with it:
virtualenv .venv/pip-tools
source .venv/pip-tools/bin/activate
pip install git+https://github.com/nvie/pip-tools.git@future
Regards,
Darragh Bailey
"Nothing is foolproof to a sufficiently talented fool" - Unknown
On 22/01/15 03:45, Joshua Harlow wrote:
A slightly better version that starts to go deeper (and downloads
dependencies of dependencies and extracts there egg_info to get at
these dependencies...)
https://gist.github.com/harlowja/555ea019aef4e901897b
Output @ http://paste.ubuntu.com/9813919/
When ran on the same 'test.txt' mentioned below...
Happy hacking!
-Josh
Joshua Harlow wrote:
A run that shows more of the happy/desired path:
$ cat test.txt
six>1
taskflow<0.5
$ python pippin.py -r test.txt
Initial package set:
- six ['>1']
- taskflow ['<0.5']
Deep package set:
- six ['==1.9.0']
- taskflow ['==0.4.0']
-Josh
Joshua Harlow wrote:
Another thing that I just started whipping together:
https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f
The idea for the above is to use pip to download dependencies, but
figure out what versions will work using our own resolver (and our own
querying of 'http://pypi.python.org/pypi/%s/json') that just does a
very
deep search of all requirements (and requirements of requirements...).
The idea for that is that the probe() function in that gist will
'freeze' a single requirement then dive down into further requirements
and ensure compatibility while that 'diving' (aka, recursion into
further requirements) is underway. If a incompatibility is found then
the recursion will back-track and try a to freeze a different
version of
a desired package (and repeat...).
To me this kind of deep finding would be a potential way of making
this
work in a way that basically only uses pip for downloading (and does
the
deep matching/probing) on our own since once the algorithm above
doesn't
backtrack and finds a matching set of requirements that will all work
together the program can exit (and this set can then be used as the
master set for openstack; at that point we might have to tell
people to
not use pip, or to only use pip --download to fetch the compatible
versions).
It's not completed but it could be complementary to what others are
working on; feel free to hack away :)
So far the following works:
$ cat test.txt
six>1
taskflow>1
$ python pippin.py -r test.txt
Initial package set:
- six ['>1']
- taskflow ['>1']
Traceback (most recent call last):
File "pippin.py", line 168, in
main()
File "pippin.py", line 162, in main
matches = probe(initial, {})
File "pippin.py", line 139, in probe
result = probe(require
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
Cool,
I've got to try that out today to see what it's doing.
I've also shoved my little program up @
https://github.com/harlowja/pippin (the pip-tools one is definitely more
elegantly coded than mine, haha).
Feel free to fork it (modify, run, or ...)
Basic instructions to use it:
https://github.com/harlowja/pippin#pippin
-Josh
Bailey, Darragh wrote:
You may find the code for pip-compile
https://github.com/nvie/pip-tools/tree/future of interest for this, as I
think they may already have a solution for the deep dependency analysis.
I've started experimenting with it for git-upstream cause GitPython have
a habbit of breaking stuff through a couple of releases now :-(
What I like is:
* Doesn't require an extra tool before using 'pip install'
** Some may want to regen the dependencies, but it's optional and the
common python dev approach is retained
* Stable releases are guaranteed to use the versions of dependencies
they were released and verified against
* Improves on the guarantee of gated branch CI
** The idea that if you sync with upstream any test failures are due to
your local changes
** Which is not always true if updated deps can break stuff
On the flip side:
* You remain exposed to security issues in python code until you
manually update
* Development cycle doesn't move forward automatically, may not see
compatibility issues until late when forced to move forward one of the deps
Think the cons can be handled by some additional CI jobs to update the
pins on a regular basis and pass it through the standard gates and
potentially to auto approve during development cycles if they pass
(already getting the latest matching ones so no big diff here). Some
decisions on trade off around whether this should be done for stable
releases automatically or periodically requiring manual approval would
have to be made.
Did I say how much I like the fact that it doesn't require another tool
before just being able to use 'pip install'?
To experiment with it:
virtualenv .venv/pip-tools
source .venv/pip-tools/bin/activate
pip install git+https://github.com/nvie/pip-tools.git@future
Regards,
Darragh Bailey
"Nothing is foolproof to a sufficiently talented fool" - Unknown
On 22/01/15 03:45, Joshua Harlow wrote:
A slightly better version that starts to go deeper (and downloads
dependencies of dependencies and extracts there egg_info to get at
these dependencies...)
https://gist.github.com/harlowja/555ea019aef4e901897b
Output @ http://paste.ubuntu.com/9813919/
When ran on the same 'test.txt' mentioned below...
Happy hacking!
-Josh
Joshua Harlow wrote:
A run that shows more of the happy/desired path:
$ cat test.txt
six>1
taskflow<0.5
$ python pippin.py -r test.txt
Initial package set:
- six ['>1']
- taskflow ['<0.5']
Deep package set:
- six ['==1.9.0']
- taskflow ['==0.4.0']
-Josh
Joshua Harlow wrote:
Another thing that I just started whipping together:
https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f
The idea for the above is to use pip to download dependencies, but
figure out what versions will work using our own resolver (and our own
querying of 'http://pypi.python.org/pypi/%s/json') that just does a
very
deep search of all requirements (and requirements of requirements...).
The idea for that is that the probe() function in that gist will
'freeze' a single requirement then dive down into further requirements
and ensure compatibility while that 'diving' (aka, recursion into
further requirements) is underway. If a incompatibility is found then
the recursion will back-track and try a to freeze a different
version of
a desired package (and repeat...).
To me this kind of deep finding would be a potential way of making this
work in a way that basically only uses pip for downloading (and does
the
deep matching/probing) on our own since once the algorithm above
doesn't
backtrack and finds a matching set of requirements that will all work
together the program can exit (and this set can then be used as the
master set for openstack; at that point we might have to tell people to
not use pip, or to only use pip --download to fetch the compatible
versions).
It's not completed but it could be complementary to what others are
working on; feel free to hack away :)
So far the following works:
$ cat test.txt
six>1
taskflow>1
$ python pippin.py -r test.txt
Initial package set:
- six ['>1']
- taskflow ['>1']
Traceback (most recent call last):
File "pippin.py", line 168, in
main()
File "pippin.py", line 162, in main
matches = probe(initial, {})
File "pippin.py", line 139, in probe
result = probe(requirements, gathered)
File "pippin.py", line 129, in probe
m = find_match(pkg_name, req)
File "pippin.py", line 112, in find_match
return match_available(req.req, find_versions(pkg_name))
File "pippin.py", line 108, in match_available
" matches '%s' (tried %s)" % (req, looked_in))
__main__.NotFoundException: No requirement found that matches
'taskflow>1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
You may find the code for pip-compile
https://github.com/nvie/pip-tools/tree/future of interest for this, as I
think they may already have a solution for the deep dependency analysis.
I've started experimenting with it for git-upstream cause GitPython have
a habbit of breaking stuff through a couple of releases now :-(
What I like is:
* Doesn't require an extra tool before using 'pip install'
** Some may want to regen the dependencies, but it's optional and the
common python dev approach is retained
* Stable releases are guaranteed to use the versions of dependencies
they were released and verified against
* Improves on the guarantee of gated branch CI
** The idea that if you sync with upstream any test failures are due to
your local changes
** Which is not always true if updated deps can break stuff
On the flip side:
* You remain exposed to security issues in python code until you
manually update
* Development cycle doesn't move forward automatically, may not see
compatibility issues until late when forced to move forward one of the deps
Think the cons can be handled by some additional CI jobs to update the
pins on a regular basis and pass it through the standard gates and
potentially to auto approve during development cycles if they pass
(already getting the latest matching ones so no big diff here). Some
decisions on trade off around whether this should be done for stable
releases automatically or periodically requiring manual approval would
have to be made.
Did I say how much I like the fact that it doesn't require another tool
before just being able to use 'pip install'?
To experiment with it:
virtualenv .venv/pip-tools
source .venv/pip-tools/bin/activate
pip install git+https://github.com/nvie/pip-tools.git@future
Regards,
Darragh Bailey
"Nothing is foolproof to a sufficiently talented fool" - Unknown
On 22/01/15 03:45, Joshua Harlow wrote:
> A slightly better version that starts to go deeper (and downloads
> dependencies of dependencies and extracts there egg_info to get at
> these dependencies...)
>
> https://gist.github.com/harlowja/555ea019aef4e901897b
>
> Output @ http://paste.ubuntu.com/9813919/
>
> When ran on the same 'test.txt' mentioned below...
>
> Happy hacking!
>
> -Josh
>
> Joshua Harlow wrote:
>> A run that shows more of the happy/desired path:
>>
>> $ cat test.txt
>> six>1
>> taskflow<0.5
>> $ python pippin.py -r test.txt
>> Initial package set:
>> - six ['>1']
>> - taskflow ['<0.5']
>> Deep package set:
>> - six ['==1.9.0']
>> - taskflow ['==0.4.0']
>>
>> -Josh
>>
>> Joshua Harlow wrote:
>>> Another thing that I just started whipping together:
>>>
>>> https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f
>>>
>>> The idea for the above is to use pip to download dependencies, but
>>> figure out what versions will work using our own resolver (and our own
>>> querying of 'http://pypi.python.org/pypi/%s/json') that just does a
>>> very
>>> deep search of all requirements (and requirements of requirements...).
>>>
>>> The idea for that is that the probe() function in that gist will
>>> 'freeze' a single requirement then dive down into further requirements
>>> and ensure compatibility while that 'diving' (aka, recursion into
>>> further requirements) is underway. If a incompatibility is found then
>>> the recursion will back-track and try a to freeze a different
>>> version of
>>> a desired package (and repeat...).
>>>
>>> To me this kind of deep finding would be a potential way of making this
>>> work in a way that basically only uses pip for downloading (and does
>>> the
>>> deep matching/probing) on our own since once the algorithm above
>>> doesn't
>>> backtrack and finds a matching set of requirements that will all work
>>> together the program can exit (and this set can then be used as the
>>> master set for openstack; at that point we might have to tell people to
>>> not use pip, or to only use pip --download to fetch the compatible
>>> versions).
>>>
>>> It's not completed but it could be complementary to what others are
>>> working on; feel free to hack away :)
>>>
>>> So far the following works:
>>>
>>> $ cat test.txt
>>> six>1
>>> taskflow>1
>>>
>>> $ python pippin.py -r test.txt
>>> Initial package set:
>>> - six ['>1']
>>> - taskflow ['>1']
>>> Traceback (most recent call last):
>>> File "pippin.py", line 168, in
>>> main()
>>> File "pippin.py", line 162, in main
>>> matches = probe(initial, {})
>>> File "pippin.py", line 139, in probe
>>> result = probe(requirements, gathered)
>>> File "pippin.py", line 129, in probe
>>> m = find_match(pkg_name, req)
>>> File "pippin.py", line 112, in find_match
>>> return match_available(req.req, find_versions(pkg_name))
>>> File "pippin.py", line 108, in match_available
>>> " matches '%s' (tried %s)" % (req, looked_in))
>>> __main__.NotFoundException: No requirement found that matches
>>> 'taskflow>1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
>>> '0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])
>>>
>>> I suspect all tha
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
On 1/15/2015 5:41 PM, Sean Dague wrote: On 01/15/2015 06:25 PM, Joe Gordon wrote: We can side step the dependency graphing and ordering issue by looking at the list of curently installed packages via pip freeze and not installing dependencies (pip install --no-deps) After looking into this further here are the known issues: * Partial capping won't work [0], so we need to pin all dependencies, we can generate this list per file via "pip install -r" and "pip freeze", but this doesn't address the issue of apt-get vs pip install. For example in the stable gate we use suds 0.4.1 but only suds 0.4.0 is available via pip. * Not all packages are installed in are standard dsvm-tempest env, so using pip-freeze from that isn't enough * We need to run this per requirements file and move to using pip install --no-deps everywhere. As the global-requirements sync wouldn't work the first time since files don't list all transient dependencies yet. So that's basically writing our own dependency system entirely, and skipping pip's (vs. fudging around pip's issues). I expect that's going to go poorly in the OpenStack ecosystem realm because a lot of very repetitive manual analysis will need to be done on each project. And if we want to bump a cap, regenerating all the graphs becomes another manual process. * We can still break if a package version is removed from pypi * in pip-freeze we sometimes install versions lower then our minimum version (python-libvirt!) That's because python-libvirt is not in any requirements.txt file, so we're taking the system version. It's in test-requirements.txt, but not for long. [1] -Sean [1] https://review.openstack.org/#/c/150148/ -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
Btw try this on 'pip<6' (for those that want to); Pip 6+ moved some of the code around that this uses; for feel to update it though and adjust it to find where the new stuff is :-P Joshua Harlow wrote: Here is one that is more complete: https://gist.github.com/harlowja/e22e8c0771b336ca392f Using a really simple requirement set that will not work: $ cat test.txt taskflow networkx<1.5 Running the above gist on that (and waiting for a while; since it does do a lot of backtracking and trying of different versions) the program will eventually break with something like the following: http://paste.ubuntu.com/9907530/ The above can likely be optimized more (to avoid diving into requirements that have already failed being selected) but it does seem to work as a satisfiability solver that will recursively check requirements deeply and try as many solutions as possible (with backtracking when a requirement is selected that doesn't work). -Josh Joshua Harlow wrote: Seems like a simple fix? https://github.com/pypa/pip/blob/1.5.6/pip/req.py#L1536 Make a new session somewhere in that gist/code and profit? Kevin L. Mitchell wrote: On Wed, 2015-01-21 at 18:48 -0800, Joshua Harlow wrote: Another thing that I just started whipping together: https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f One problem, though, is that parse_requirements() now requires the session keyword argument. In version 6.0.6, parse_requirements() begins with: def parse_requirements(filename, finder=None, comes_from=None, options=None, session=None): if session is None: raise TypeError( "parse_requirements() missing 1 required keyword argument: " "'session'" ) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
Here is one that is more complete: https://gist.github.com/harlowja/e22e8c0771b336ca392f Using a really simple requirement set that will not work: $ cat test.txt taskflow networkx<1.5 Running the above gist on that (and waiting for a while; since it does do a lot of backtracking and trying of different versions) the program will eventually break with something like the following: http://paste.ubuntu.com/9907530/ The above can likely be optimized more (to avoid diving into requirements that have already failed being selected) but it does seem to work as a satisfiability solver that will recursively check requirements deeply and try as many solutions as possible (with backtracking when a requirement is selected that doesn't work). -Josh Joshua Harlow wrote: Seems like a simple fix? https://github.com/pypa/pip/blob/1.5.6/pip/req.py#L1536 Make a new session somewhere in that gist/code and profit? Kevin L. Mitchell wrote: On Wed, 2015-01-21 at 18:48 -0800, Joshua Harlow wrote: Another thing that I just started whipping together: https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f One problem, though, is that parse_requirements() now requires the session keyword argument. In version 6.0.6, parse_requirements() begins with: def parse_requirements(filename, finder=None, comes_from=None, options=None, session=None): if session is None: raise TypeError( "parse_requirements() missing 1 required keyword argument: " "'session'" ) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
Seems like a simple fix? https://github.com/pypa/pip/blob/1.5.6/pip/req.py#L1536 Make a new session somewhere in that gist/code and profit? Kevin L. Mitchell wrote: On Wed, 2015-01-21 at 18:48 -0800, Joshua Harlow wrote: Another thing that I just started whipping together: https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f One problem, though, is that parse_requirements() now requires the session keyword argument. In version 6.0.6, parse_requirements() begins with: def parse_requirements(filename, finder=None, comes_from=None, options=None, session=None): if session is None: raise TypeError( "parse_requirements() missing 1 required keyword argument: " "'session'" ) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
On Wed, 2015-01-21 at 18:48 -0800, Joshua Harlow wrote: > Another thing that I just started whipping together: > > https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f One problem, though, is that parse_requirements() now requires the session keyword argument. In version 6.0.6, parse_requirements() begins with: def parse_requirements(filename, finder=None, comes_from=None, options=None, session=None): if session is None: raise TypeError( "parse_requirements() missing 1 required keyword argument: " "'session'" ) -- Kevin L. Mitchell Rackspace __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
A slightly better version that starts to go deeper (and downloads
dependencies of dependencies and extracts there egg_info to get at these
dependencies...)
https://gist.github.com/harlowja/555ea019aef4e901897b
Output @ http://paste.ubuntu.com/9813919/
When ran on the same 'test.txt' mentioned below...
Happy hacking!
-Josh
Joshua Harlow wrote:
A run that shows more of the happy/desired path:
$ cat test.txt
six>1
taskflow<0.5
$ python pippin.py -r test.txt
Initial package set:
- six ['>1']
- taskflow ['<0.5']
Deep package set:
- six ['==1.9.0']
- taskflow ['==0.4.0']
-Josh
Joshua Harlow wrote:
Another thing that I just started whipping together:
https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f
The idea for the above is to use pip to download dependencies, but
figure out what versions will work using our own resolver (and our own
querying of 'http://pypi.python.org/pypi/%s/json') that just does a very
deep search of all requirements (and requirements of requirements...).
The idea for that is that the probe() function in that gist will
'freeze' a single requirement then dive down into further requirements
and ensure compatibility while that 'diving' (aka, recursion into
further requirements) is underway. If a incompatibility is found then
the recursion will back-track and try a to freeze a different version of
a desired package (and repeat...).
To me this kind of deep finding would be a potential way of making this
work in a way that basically only uses pip for downloading (and does the
deep matching/probing) on our own since once the algorithm above doesn't
backtrack and finds a matching set of requirements that will all work
together the program can exit (and this set can then be used as the
master set for openstack; at that point we might have to tell people to
not use pip, or to only use pip --download to fetch the compatible
versions).
It's not completed but it could be complementary to what others are
working on; feel free to hack away :)
So far the following works:
$ cat test.txt
six>1
taskflow>1
$ python pippin.py -r test.txt
Initial package set:
- six ['>1']
- taskflow ['>1']
Traceback (most recent call last):
File "pippin.py", line 168, in
main()
File "pippin.py", line 162, in main
matches = probe(initial, {})
File "pippin.py", line 139, in probe
result = probe(requirements, gathered)
File "pippin.py", line 129, in probe
m = find_match(pkg_name, req)
File "pippin.py", line 112, in find_match
return match_available(req.req, find_versions(pkg_name))
File "pippin.py", line 108, in match_available
" matches '%s' (tried %s)" % (req, looked_in))
__main__.NotFoundException: No requirement found that matches
'taskflow>1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
'0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])
I suspect all that is needed to add is the code that is marked with
FIXME/TODO there and this kind of recursive back-tracking might just do
the trick...
-Josh
Joe Gordon wrote:
On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon mailto:joe.gord...@gmail.com>> wrote:
We can side step the dependency graphing and ordering issue by
looking at the list of curently installed packages via pip freeze
and not installing dependencies (pip install --no-deps)
After looking into this further here are the known issues:
* Partial capping won't work [0], so we need to pin all
dependencies, we can generate this list per file via "pip install
-r" and "pip freeze", but this doesn't address the issue of apt-get
vs pip install. For example in the stable gate we use suds 0.4.1 but
only suds 0.4.0 is available via pip.
* Not all packages are installed in are standard dsvm-tempest env,
so using pip-freeze from that isn't enough
* We need to run this per requirements file and move to using pip
install --no-deps everywhere. As the global-requirements sync
wouldn't work the first time since files don't list all transient
dependencies yet.
* We can still break if a package version is removed from pypi
* in pip-freeze we sometimes install versions lower then our minimum
version (python-libvirt!)
Exploring a few ideas here: https://review.openstack.org/#/c/147451/4
[0]
http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html
On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley mailto:fu...@yuggoth.org>> wrote:
On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
[...]
> The other thing that happened was partial capping doesn't work,
> because something else moves forward and breaks you from below. So
> the patch will need to hit everything at once.
Right, and we _have_ to start using stable branches on all
clients/libraries to backport fixes as part of that. This means that
the stable branch management workflow is about to become pervasive
across some teams who were previously (blissfully?) ignorant of it.
> Unresolved entirely is the tertiary dependencies (not direct
> dependencies of any OpenStack project). That will need another
> mechanism to seed them before any in
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
A run that shows more of the happy/desired path:
$ cat test.txt
six>1
taskflow<0.5
$ python pippin.py -r test.txt
Initial package set:
- six ['>1']
- taskflow ['<0.5']
Deep package set:
- six ['==1.9.0']
- taskflow ['==0.4.0']
-Josh
Joshua Harlow wrote:
Another thing that I just started whipping together:
https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f
The idea for the above is to use pip to download dependencies, but
figure out what versions will work using our own resolver (and our own
querying of 'http://pypi.python.org/pypi/%s/json') that just does a very
deep search of all requirements (and requirements of requirements...).
The idea for that is that the probe() function in that gist will
'freeze' a single requirement then dive down into further requirements
and ensure compatibility while that 'diving' (aka, recursion into
further requirements) is underway. If a incompatibility is found then
the recursion will back-track and try a to freeze a different version of
a desired package (and repeat...).
To me this kind of deep finding would be a potential way of making this
work in a way that basically only uses pip for downloading (and does the
deep matching/probing) on our own since once the algorithm above doesn't
backtrack and finds a matching set of requirements that will all work
together the program can exit (and this set can then be used as the
master set for openstack; at that point we might have to tell people to
not use pip, or to only use pip --download to fetch the compatible
versions).
It's not completed but it could be complementary to what others are
working on; feel free to hack away :)
So far the following works:
$ cat test.txt
six>1
taskflow>1
$ python pippin.py -r test.txt
Initial package set:
- six ['>1']
- taskflow ['>1']
Traceback (most recent call last):
File "pippin.py", line 168, in
main()
File "pippin.py", line 162, in main
matches = probe(initial, {})
File "pippin.py", line 139, in probe
result = probe(requirements, gathered)
File "pippin.py", line 129, in probe
m = find_match(pkg_name, req)
File "pippin.py", line 112, in find_match
return match_available(req.req, find_versions(pkg_name))
File "pippin.py", line 108, in match_available
" matches '%s' (tried %s)" % (req, looked_in))
__main__.NotFoundException: No requirement found that matches
'taskflow>1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
'0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])
I suspect all that is needed to add is the code that is marked with
FIXME/TODO there and this kind of recursive back-tracking might just do
the trick...
-Josh
Joe Gordon wrote:
On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon mailto:joe.gord...@gmail.com>> wrote:
We can side step the dependency graphing and ordering issue by
looking at the list of curently installed packages via pip freeze
and not installing dependencies (pip install --no-deps)
After looking into this further here are the known issues:
* Partial capping won't work [0], so we need to pin all
dependencies, we can generate this list per file via "pip install
-r" and "pip freeze", but this doesn't address the issue of apt-get
vs pip install. For example in the stable gate we use suds 0.4.1 but
only suds 0.4.0 is available via pip.
* Not all packages are installed in are standard dsvm-tempest env,
so using pip-freeze from that isn't enough
* We need to run this per requirements file and move to using pip
install --no-deps everywhere. As the global-requirements sync
wouldn't work the first time since files don't list all transient
dependencies yet.
* We can still break if a package version is removed from pypi
* in pip-freeze we sometimes install versions lower then our minimum
version (python-libvirt!)
Exploring a few ideas here: https://review.openstack.org/#/c/147451/4
[0]
http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html
On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley mailto:fu...@yuggoth.org>> wrote:
On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
[...]
> The other thing that happened was partial capping doesn't work,
> because something else moves forward and breaks you from below. So
> the patch will need to hit everything at once.
Right, and we _have_ to start using stable branches on all
clients/libraries to backport fixes as part of that. This means that
the stable branch management workflow is about to become pervasive
across some teams who were previously (blissfully?) ignorant of it.
> Unresolved entirely is the tertiary dependencies (not direct
> dependencies of any OpenStack project). That will need another
> mechanism to seed them before any installation happens.
[...]
I won't go so far as to call it intractable, but I took a stab at it
about a year ago and building the dependency graph properly to be
able to do a depth-first ordering is nontrivial (enough that after
about a week hacking on possible solutions I gave up and switched to
more productive tasks). The primary complications I ran
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
Another thing that I just started whipping together:
https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f
The idea for the above is to use pip to download dependencies, but
figure out what versions will work using our own resolver (and our own
querying of 'http://pypi.python.org/pypi/%s/json') that just does a very
deep search of all requirements (and requirements of requirements...).
The idea for that is that the probe() function in that gist will
'freeze' a single requirement then dive down into further requirements
and ensure compatibility while that 'diving' (aka, recursion into
further requirements) is underway. If a incompatibility is found then
the recursion will back-track and try a to freeze a different version of
a desired package (and repeat...).
To me this kind of deep finding would be a potential way of making this
work in a way that basically only uses pip for downloading (and does the
deep matching/probing) on our own since once the algorithm above doesn't
backtrack and finds a matching set of requirements that will all work
together the program can exit (and this set can then be used as the
master set for openstack; at that point we might have to tell people to
not use pip, or to only use pip --download to fetch the compatible
versions).
It's not completed but it could be complementary to what others are
working on; feel free to hack away :)
So far the following works:
$ cat test.txt
six>1
taskflow>1
$ python pippin.py -r test.txt
Initial package set:
- six ['>1']
- taskflow ['>1']
Traceback (most recent call last):
File "pippin.py", line 168, in
main()
File "pippin.py", line 162, in main
matches = probe(initial, {})
File "pippin.py", line 139, in probe
result = probe(requirements, gathered)
File "pippin.py", line 129, in probe
m = find_match(pkg_name, req)
File "pippin.py", line 112, in find_match
return match_available(req.req, find_versions(pkg_name))
File "pippin.py", line 108, in match_available
" matches '%s' (tried %s)" % (req, looked_in))
__main__.NotFoundException: No requirement found that matches
'taskflow>1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
'0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])
I suspect all that is needed to add is the code that is marked with
FIXME/TODO there and this kind of recursive back-tracking might just do
the trick...
-Josh
Joe Gordon wrote:
On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon mailto:joe.gord...@gmail.com>> wrote:
We can side step the dependency graphing and ordering issue by
looking at the list of curently installed packages via pip freeze
and not installing dependencies (pip install --no-deps)
After looking into this further here are the known issues:
* Partial capping won't work [0], so we need to pin all
dependencies, we can generate this list per file via "pip install
-r" and "pip freeze", but this doesn't address the issue of apt-get
vs pip install. For example in the stable gate we use suds 0.4.1 but
only suds 0.4.0 is available via pip.
* Not all packages are installed in are standard dsvm-tempest env,
so using pip-freeze from that isn't enough
* We need to run this per requirements file and move to using pip
install --no-deps everywhere. As the global-requirements sync
wouldn't work the first time since files don't list all transient
dependencies yet.
* We can still break if a package version is removed from pypi
* in pip-freeze we sometimes install versions lower then our minimum
version (python-libvirt!)
Exploring a few ideas here: https://review.openstack.org/#/c/147451/4
[0]
http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html
On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley mailto:fu...@yuggoth.org>> wrote:
On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
[...]
> The other thing that happened was partial capping doesn't work,
> because something else moves forward and breaks you from below. So
> the patch will need to hit everything at once.
Right, and we _have_ to start using stable branches on all
clients/libraries to backport fixes as part of that. This means that
the stable branch management workflow is about to become pervasive
across some teams who were previously (blissfully?) ignorant of it.
> Unresolved entirely is the tertiary dependencies (not direct
> dependencies of any OpenStack project). That will need another
> mechanism to seed them before any installation happens.
[...]
I won't go so far as to call it intractable, but I took a stab at it
about a year ago and building the dependency graph properly to be
able to do a depth-first ordering is nontrivial (enough that after
about a week hacking on possible solutions I gave up and switched to
more productive tas
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
On 15/01/15 23:41, Sean Dague wrote: > On 01/15/2015 06:25 PM, Joe Gordon wrote: >> We can side step the dependency graphing and ordering issue by looking >> at the list of curently installed packages via pip freeze and not >> installing dependencies (pip install --no-deps) >> >> After looking into this further here are the known issues: >> >> * Partial capping won't work [0], so we need to pin all dependencies, we >> can generate this list per file via "pip install -r" and "pip freeze", >> but this doesn't address the issue of apt-get vs pip install. For >> example in the stable gate we use suds 0.4.1 but only suds 0.4.0 is >> available via pip. >> * Not all packages are installed in are standard dsvm-tempest env, so >> using pip-freeze from that isn't enough >> * We need to run this per requirements file and move to using pip >> install --no-deps everywhere. As the global-requirements sync wouldn't >> work the first time since files don't list all transient dependencies yet. > So that's basically writing our own dependency system entirely, and > skipping pip's (vs. fudging around pip's issues). I expect that's going > to go poorly in the OpenStack ecosystem realm because a lot of very > repetitive manual analysis will need to be done on each project. And if > we want to bump a cap, regenerating all the graphs becomes another > manual process. Pip's loose dependency processing where each entry is treated as an entirely separate resolution with no consideration of previous or later constraints came up recently in a discussion at work. Looking around I discovered someone had already started working on a solution, which involves moving the current requirements to a 'requirements.in' file and then compiling that to requirements.txt, which would contain only fully pinned packages in the correct order for pip to install them without needing to add any special options. Might be worth seeing if that could be developed further: http://nvie.com/posts/better-package-management/ https://github.com/nvie/pip-tools/tree/future >> * We can still break if a package version is removed from pypi >> * in pip-freeze we sometimes install versions lower then our minimum >> version (python-libvirt!) > That's because python-libvirt is not in any requirements.txt file, so > we're taking the system version. > > -Sean > Regards, Darragh Bailey "Nothing is foolproof to a sufficiently talented fool" - Unknown __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
On 01/15/2015 06:25 PM, Joe Gordon wrote: > We can side step the dependency graphing and ordering issue by looking > at the list of curently installed packages via pip freeze and not > installing dependencies (pip install --no-deps) > > After looking into this further here are the known issues: > > * Partial capping won't work [0], so we need to pin all dependencies, we > can generate this list per file via "pip install -r" and "pip freeze", > but this doesn't address the issue of apt-get vs pip install. For > example in the stable gate we use suds 0.4.1 but only suds 0.4.0 is > available via pip. > * Not all packages are installed in are standard dsvm-tempest env, so > using pip-freeze from that isn't enough > * We need to run this per requirements file and move to using pip > install --no-deps everywhere. As the global-requirements sync wouldn't > work the first time since files don't list all transient dependencies yet. So that's basically writing our own dependency system entirely, and skipping pip's (vs. fudging around pip's issues). I expect that's going to go poorly in the OpenStack ecosystem realm because a lot of very repetitive manual analysis will need to be done on each project. And if we want to bump a cap, regenerating all the graphs becomes another manual process. > * We can still break if a package version is removed from pypi > * in pip-freeze we sometimes install versions lower then our minimum > version (python-libvirt!) That's because python-libvirt is not in any requirements.txt file, so we're taking the system version. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon wrote: > We can side step the dependency graphing and ordering issue by looking at > the list of curently installed packages via pip freeze and not installing > dependencies (pip install --no-deps) > > After looking into this further here are the known issues: > > * Partial capping won't work [0], so we need to pin all dependencies, we > can generate this list per file via "pip install -r" and "pip freeze", but > this doesn't address the issue of apt-get vs pip install. For example in > the stable gate we use suds 0.4.1 but only suds 0.4.0 is available via pip. > * Not all packages are installed in are standard dsvm-tempest env, so > using pip-freeze from that isn't enough > * We need to run this per requirements file and move to using pip install > --no-deps everywhere. As the global-requirements sync wouldn't work the > first time since files don't list all transient dependencies yet. > * We can still break if a package version is removed from pypi > * in pip-freeze we sometimes install versions lower then our minimum > version (python-libvirt!) > > Exploring a few ideas here: https://review.openstack.org/#/c/147451/4 > > > [0] > http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html > > On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley wrote: > >> On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote: >> [...] >> > The other thing that happened was partial capping doesn't work, >> > because something else moves forward and breaks you from below. So >> > the patch will need to hit everything at once. >> >> Right, and we _have_ to start using stable branches on all >> clients/libraries to backport fixes as part of that. This means that >> the stable branch management workflow is about to become pervasive >> across some teams who were previously (blissfully?) ignorant of it. >> >> > Unresolved entirely is the tertiary dependencies (not direct >> > dependencies of any OpenStack project). That will need another >> > mechanism to seed them before any installation happens. >> [...] >> >> I won't go so far as to call it intractable, but I took a stab at it >> about a year ago and building the dependency graph properly to be >> able to do a depth-first ordering is nontrivial (enough that after >> about a week hacking on possible solutions I gave up and switched to >> more productive tasks). The primary complications I ran into were >> identifying setup_requires in transitive dependencies and dealing >> with platform/version-specific dependencies. That said, there's a >> very good chance that more recent improvements in setuptools, pip >> and virtualenv could make this task easier. >> >> > That's the things I can think off from the top of my head. >> >> The implementation, from a devstack-gate perspective, is also going >> to require a decision on whether we stick with stable/relname for >> branches of libraries too or switch to some extended branch mapping >> mechanism to be able to track stable/relnum branches for those. And >> we're going to need more jobs to ensure that clients (specifically) >> retain backward-compatibility from an appdev and end user >> perspective since they'll no longer get any testing as server >> dependencies on stable branches (due to being capped there). >> -- >> Jeremy Stanley >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
We can side step the dependency graphing and ordering issue by looking at the list of curently installed packages via pip freeze and not installing dependencies (pip install --no-deps) After looking into this further here are the known issues: * Partial capping won't work [0], so we need to pin all dependencies, we can generate this list per file via "pip install -r" and "pip freeze", but this doesn't address the issue of apt-get vs pip install. For example in the stable gate we use suds 0.4.1 but only suds 0.4.0 is available via pip. * Not all packages are installed in are standard dsvm-tempest env, so using pip-freeze from that isn't enough * We need to run this per requirements file and move to using pip install --no-deps everywhere. As the global-requirements sync wouldn't work the first time since files don't list all transient dependencies yet. * We can still break if a package version is removed from pypi * in pip-freeze we sometimes install versions lower then our minimum version (python-libvirt!) [0] http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley wrote: > On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote: > [...] > > The other thing that happened was partial capping doesn't work, > > because something else moves forward and breaks you from below. So > > the patch will need to hit everything at once. > > Right, and we _have_ to start using stable branches on all > clients/libraries to backport fixes as part of that. This means that > the stable branch management workflow is about to become pervasive > across some teams who were previously (blissfully?) ignorant of it. > > > Unresolved entirely is the tertiary dependencies (not direct > > dependencies of any OpenStack project). That will need another > > mechanism to seed them before any installation happens. > [...] > > I won't go so far as to call it intractable, but I took a stab at it > about a year ago and building the dependency graph properly to be > able to do a depth-first ordering is nontrivial (enough that after > about a week hacking on possible solutions I gave up and switched to > more productive tasks). The primary complications I ran into were > identifying setup_requires in transitive dependencies and dealing > with platform/version-specific dependencies. That said, there's a > very good chance that more recent improvements in setuptools, pip > and virtualenv could make this task easier. > > > That's the things I can think off from the top of my head. > > The implementation, from a devstack-gate perspective, is also going > to require a decision on whether we stick with stable/relname for > branches of libraries too or switch to some extended branch mapping > mechanism to be able to track stable/relnum branches for those. And > we're going to need more jobs to ensure that clients (specifically) > retain backward-compatibility from an appdev and end user > perspective since they'll no longer get any testing as server > dependencies on stable branches (due to being capped there). > -- > Jeremy Stanley > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote: [...] > The other thing that happened was partial capping doesn't work, > because something else moves forward and breaks you from below. So > the patch will need to hit everything at once. Right, and we _have_ to start using stable branches on all clients/libraries to backport fixes as part of that. This means that the stable branch management workflow is about to become pervasive across some teams who were previously (blissfully?) ignorant of it. > Unresolved entirely is the tertiary dependencies (not direct > dependencies of any OpenStack project). That will need another > mechanism to seed them before any installation happens. [...] I won't go so far as to call it intractable, but I took a stab at it about a year ago and building the dependency graph properly to be able to do a depth-first ordering is nontrivial (enough that after about a week hacking on possible solutions I gave up and switched to more productive tasks). The primary complications I ran into were identifying setup_requires in transitive dependencies and dealing with platform/version-specific dependencies. That said, there's a very good chance that more recent improvements in setuptools, pip and virtualenv could make this task easier. > That's the things I can think off from the top of my head. The implementation, from a devstack-gate perspective, is also going to require a decision on whether we stick with stable/relname for branches of libraries too or switch to some extended branch mapping mechanism to be able to track stable/relnum branches for those. And we're going to need more jobs to ensure that clients (specifically) retain backward-compatibility from an appdev and end user perspective since they'll no longer get any testing as server dependencies on stable branches (due to being capped there). -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
On 01/15/2015 08:38 AM, Thierry Carrez wrote: > Sean Dague wrote: >> The policy decision to cap stable requirements was agreed to previously >> (and at summit this year), it just needs someone to implement. > > +1 > > We started to implement it (in Oslo libraries iirc) and then reverted > because we encountered problems. We concluded a proper cross-project > spec should be written so that we can try to anticipate issues better. > > I don't have a totally clear view of the issue(s) we encoutered last > time, though. Could someone write a postmortem or point me to one ? Or > was the upgrading components issue (fixed in > https://review.openstack.org/#/c/142075/) the only problem ? My recollection is the component upgrade order was a big problem, but not the only one. The other thing that happened was partial capping doesn't work, because something else moves forward and breaks you from below. So the patch will need to hit everything at once. Unresolved entirely is the tertiary dependencies (not direct dependencies of any OpenStack project). That will need another mechanism to seed them before any installation happens. Whether or not pip should be pinned is also a good question, especially because pbr requires pip, so the install -U on pbr tends to uplift pip even if it was previously downgraded. That's the things I can think off from the top of my head. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
Sean Dague wrote: > The policy decision to cap stable requirements was agreed to previously > (and at summit this year), it just needs someone to implement. +1 We started to implement it (in Oslo libraries iirc) and then reverted because we encountered problems. We concluded a proper cross-project spec should be written so that we can try to anticipate issues better. I don't have a totally clear view of the issue(s) we encoutered last time, though. Could someone write a postmortem or point me to one ? Or was the upgrading components issue (fixed in https://review.openstack.org/#/c/142075/) the only problem ? -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
On 01/15/2015 05:21 AM, Nikola Đipanov wrote: > On 01/15/2015 10:35 AM, Joe Gordon wrote: > >> So how could we have avoided this problem? By capping stable branch >> requirements so we only have to worry about uncapped dependencies on >> master. Capping stable branches has been previous discussed but no >> action has been taken. So going forward I propose we pin all >> requirements, including transitive, on stable branches. This way the >> release of new dependencies cannot automatically break stable branches >> and thus break grenade on master. >> > > This is an absolute must IMHO, including transitive dependencies, > because if they are not capped - they can cause other issues like bring > in additional deps a stable release is not even supposed to have, among > all the usual issues. > > The problem as I understand it is that this breaks how we do upgrades > testing in the gate, AKA "the granade job" (all in a single VM, install > everything from pip). IMHO this is broken and needs to be fixed ASAP, if > capping breaks it. That's a mis understanding of the last issue, grenade handles this fine. The capping only caused a problem because we were upgrading components in an incorrect issue: Swift upgrading before Ceilometer, even though Ceilometer had middleware installed in Swift. Which meant that for almost ever we were actually running new Swift with old Ceilometer middleware. That was fixed here - https://review.openstack.org/#/c/142075/ The policy decision to cap stable requirements was agreed to previously (and at summit this year), it just needs someone to implement. This also came up in this keystoneclient thread from last week - http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html Volunteers welcomed. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
On 01/15/2015 10:35 AM, Joe Gordon wrote: > So how could we have avoided this problem? By capping stable branch > requirements so we only have to worry about uncapped dependencies on > master. Capping stable branches has been previous discussed but no > action has been taken. So going forward I propose we pin all > requirements, including transitive, on stable branches. This way the > release of new dependencies cannot automatically break stable branches > and thus break grenade on master. > This is an absolute must IMHO, including transitive dependencies, because if they are not capped - they can cause other issues like bring in additional deps a stable release is not even supposed to have, among all the usual issues. The problem as I understand it is that this breaks how we do upgrades testing in the gate, AKA "the granade job" (all in a single VM, install everything from pip). IMHO this is broken and needs to be fixed ASAP, if capping breaks it. N. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
> capping stable branch requirements so we only have to worry about uncapped > dependencies on master +1 this is a sensible idea not just from the POV of grenade testing, but general stability of the stable releases. As per Joe's link [5] we can see 3rd party libraries may make breaking changes in minor releases, so we should pin to the exact version (ie ==X.Y.Z, rather than saying we'll allow increases in the Z for bugfixes). > how about the deep dependency? We should pin transitive dependencies too. If we don't then we're open to new 3rd party releases breaking things just as before. mg On Thu, Jan 15, 2015 at 9:51 AM, ZhiQiang Fan wrote: > +1 > > how about the deep dependency? for example, we depends package A, and pin > it, but A->B->C, then B and C are not pined since they are not directly > depended, then what should we do? pin everything? > > On Thu, Jan 15, 2015 at 5:35 PM, Joe Gordon wrote: >> >> Eventlet released 0.16.1 on 2015-01-14 [0], which removed a deprecated API >> that nova stable/* still used. This caused nova-compute in stable/juno and >> stable/icehouse to crash thus breaking grenade on master. In 24 hours this >> bug caused 671 grenade jobs to fail[1]! >> >> After some quick debugging of this cryptic error suppressing stacktrace >> [2] we got to the real stacktrace: >> " File "nova/virt/libvirt/__init__.py", line 15, in >> >> from nova.virt.libvirt import driver >> File "nova/virt/libvirt/driver.py", line 48, in >> from eventlet import util as eventlet_util >> >> ImportError: cannot import name util" [3] >> and capped the eventlet version on stable/* [4] >> >> As dependency bugs go this was a pretty typical one, so why am I writing >> this? Because we knew about this bug before it hit the gate, yet it was >> still an issue. The util module was removed in 0.16.0 but 'sneaked into' the >> build [5] so 0.16.1 fixed that. Before 0.16.1 was released this bug only >> impacted people who installed eventlet 0.16.0 from source and not from pip. >> Luckily someone did this and filed a bug [6] and a fix for this was landed >> on master on January 7th, and by the 10th the fix was backported to >> stable/juno and ready [7]. In the mean time, we had an unusually bad week >> dealing with new library versions; boto 2.35.0 was released and broke master >> and stable/juno [8] and sqlalchemy-migrate broke glance on master and stable >> as well [9]. And by the time these issues were fixed, and the tests would >> pass, eventlet 0.16.1 was already out. Unfortunately, there are a very small >> number of people who work on fixing dependency issues, some of them were >> traveling and the rest were over worked, so figuring out what went wrong >> with all the new packages came down to a handful of overworked people. >> >> So how could we have avoided this problem? By capping stable branch >> requirements so we only have to worry about uncapped dependencies on master. >> Capping stable branches has been previous discussed but no action has been >> taken. So going forward I propose we pin all requirements, including >> transitive, on stable branches. This way the release of new dependencies >> cannot automatically break stable branches and thus break grenade on master. >> >> I think we can implement this using `pip freeze` to get a list of all the >> installed libraries and `pip install -r --no-deps` to install the specific >> package versions. I am not sure how best to handle package versions being >> removed from pypi though. Or at the very least we can cap requirements in >> global requirements and at least reduce the number of packages we install >> uncapped versions of. >> >> Thoughts? >> >> best, >> Joe >> >> [0] https://pypi.python.org/pypi/eventlet/0.16.1 >> [1] http://status.openstack.org/elastic-recheck/#1410626 >> [2] >> http://logs.openstack.org/98/115998/59/check/check-grenade-dsvm/e57eb40/logs/old/screen-n-cpu.txt.gz >> [3] http://paste.openstack.org/show/157558/ >> [4] >> https://review.openstack.org/#/q/I4bbbeb5bf9c22ed36f5c9a74fec6b487d2c15697,n,z >> [5] https://github.com/eventlet/eventlet/releases/tag/v0.16.1 >> [6] https://bugs.launchpad.net/nova/+bug/1407685 >> [7] >> https://review.openstack.org/#/q/Idbb9d2b53829dae0e807cd1260dee3dce155d5f3,n,z >> [8] https://bugs.launchpad.net/nova/+bug/1408987 >> [9] https://bugs.launchpad.net/glance/+bug/1410494 >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > > -- > blog: zqfan.github.com > git: github.com/zqfan > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/o
Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
+1 how about the deep dependency? for example, we depends package A, and pin it, but A->B->C, then B and C are not pined since they are not directly depended, then what should we do? pin everything? On Thu, Jan 15, 2015 at 5:35 PM, Joe Gordon wrote: > Eventlet released 0.16.1 on 2015-01-14 [0], which removed a deprecated > API that nova stable/* still used. This caused nova-compute in stable/juno > and stable/icehouse to crash thus breaking grenade on master. In 24 hours > this bug caused 671 grenade jobs to fail[1]! > > After some quick debugging of this cryptic error suppressing stacktrace > [2] we got to the real stacktrace: > " File "nova/virt/libvirt/__init__.py", line 15, in > > from nova.virt.libvirt import driver > File "nova/virt/libvirt/driver.py", line 48, in > from eventlet import util as eventlet_util > > ImportError: cannot import name util" [3] > and capped the eventlet version on stable/* [4] > > As dependency bugs go this was a pretty typical one, so why am I writing > this? Because we knew about this bug before it hit the gate, yet it was > still an issue. The util module was removed in 0.16.0 but 'sneaked into' > the build [5] so 0.16.1 fixed that. Before 0.16.1 was released this bug > only impacted people who installed eventlet 0.16.0 from source and not from > pip. Luckily someone did this and filed a bug [6] and a fix for this was > landed on master on January 7th, and by the 10th the fix was backported to > stable/juno and ready [7]. In the mean time, we had an unusually bad week > dealing with new library versions; boto 2.35.0 was released and broke > master and stable/juno [8] and sqlalchemy-migrate broke glance on master > and stable as well [9]. And by the time these issues were fixed, and the > tests would pass, eventlet 0.16.1 was already out. Unfortunately, there are > a very small number of people who work on fixing dependency issues, some of > them were traveling and the rest were over worked, so figuring out what > went wrong with all the new packages came down to a handful of overworked > people. > > So how could we have avoided this problem? By capping stable branch > requirements so we only have to worry about uncapped dependencies on > master. Capping stable branches has been previous discussed but no action > has been taken. So going forward I propose we pin all requirements, > including transitive, on stable branches. This way the release of new > dependencies cannot automatically break stable branches and thus break > grenade on master. > > I think we can implement this using `pip freeze` to get a list of all the > installed libraries and `pip install -r --no-deps` to install the specific > package versions. I am not sure how best to handle package versions being > removed from pypi though. Or at the very least we can cap requirements in > global requirements and at least reduce the number of packages we install > uncapped versions of. > > Thoughts? > > best, > Joe > > [0] https://pypi.python.org/pypi/eventlet/0.16.1 > [1] http://status.openstack.org/elastic-recheck/#1410626 > [2] > http://logs.openstack.org/98/115998/59/check/check-grenade-dsvm/e57eb40/logs/old/screen-n-cpu.txt.gz > [3] http://paste.openstack.org/show/157558/ > [4] > https://review.openstack.org/#/q/I4bbbeb5bf9c22ed36f5c9a74fec6b487d2c15697,n,z > [5] https://github.com/eventlet/eventlet/releases/tag/v0.16.1 > [6] https://bugs.launchpad.net/nova/+bug/1407685 > [7] > https://review.openstack.org/#/q/Idbb9d2b53829dae0e807cd1260dee3dce155d5f3,n,z > [8] https://bugs.launchpad.net/nova/+bug/1408987 > [9] https://bugs.launchpad.net/glance/+bug/1410494 > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- blog: zqfan.github.com git: github.com/zqfan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate
Eventlet released 0.16.1 on 2015-01-14 [0], which removed a deprecated API that nova stable/* still used. This caused nova-compute in stable/juno and stable/icehouse to crash thus breaking grenade on master. In 24 hours this bug caused 671 grenade jobs to fail[1]! After some quick debugging of this cryptic error suppressing stacktrace [2] we got to the real stacktrace: " File "nova/virt/libvirt/__init__.py", line 15, in from nova.virt.libvirt import driver File "nova/virt/libvirt/driver.py", line 48, in from eventlet import util as eventlet_util ImportError: cannot import name util" [3] and capped the eventlet version on stable/* [4] As dependency bugs go this was a pretty typical one, so why am I writing this? Because we knew about this bug before it hit the gate, yet it was still an issue. The util module was removed in 0.16.0 but 'sneaked into' the build [5] so 0.16.1 fixed that. Before 0.16.1 was released this bug only impacted people who installed eventlet 0.16.0 from source and not from pip. Luckily someone did this and filed a bug [6] and a fix for this was landed on master on January 7th, and by the 10th the fix was backported to stable/juno and ready [7]. In the mean time, we had an unusually bad week dealing with new library versions; boto 2.35.0 was released and broke master and stable/juno [8] and sqlalchemy-migrate broke glance on master and stable as well [9]. And by the time these issues were fixed, and the tests would pass, eventlet 0.16.1 was already out. Unfortunately, there are a very small number of people who work on fixing dependency issues, some of them were traveling and the rest were over worked, so figuring out what went wrong with all the new packages came down to a handful of overworked people. So how could we have avoided this problem? By capping stable branch requirements so we only have to worry about uncapped dependencies on master. Capping stable branches has been previous discussed but no action has been taken. So going forward I propose we pin all requirements, including transitive, on stable branches. This way the release of new dependencies cannot automatically break stable branches and thus break grenade on master. I think we can implement this using `pip freeze` to get a list of all the installed libraries and `pip install -r --no-deps` to install the specific package versions. I am not sure how best to handle package versions being removed from pypi though. Or at the very least we can cap requirements in global requirements and at least reduce the number of packages we install uncapped versions of. Thoughts? best, Joe [0] https://pypi.python.org/pypi/eventlet/0.16.1 [1] http://status.openstack.org/elastic-recheck/#1410626 [2] http://logs.openstack.org/98/115998/59/check/check-grenade-dsvm/e57eb40/logs/old/screen-n-cpu.txt.gz [3] http://paste.openstack.org/show/157558/ [4] https://review.openstack.org/#/q/I4bbbeb5bf9c22ed36f5c9a74fec6b487d2c15697,n,z [5] https://github.com/eventlet/eventlet/releases/tag/v0.16.1 [6] https://bugs.launchpad.net/nova/+bug/1407685 [7] https://review.openstack.org/#/q/Idbb9d2b53829dae0e807cd1260dee3dce155d5f3,n,z [8] https://bugs.launchpad.net/nova/+bug/1408987 [9] https://bugs.launchpad.net/glance/+bug/1410494 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
