Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-31 Thread Joshua Harlow
For those that have been following this escapade I wrote up the 
following, hopefully it's useful to someone ;)


https://github.com/harlowja/pippin#how-it-works

Example(s) of actual runs are also @

https://github.com/harlowja/pippin/tree/master/examples

^ Does not include the full output files (since it's a lot of 
information/data that wouldn't fit easily on git...)


-Josh

Joshua Harlow wrote:

Cool,

I've got to try that out today to see what it's doing.

I've also shoved my little program up @
https://github.com/harlowja/pippin (the pip-tools one is definitely more
elegantly coded than mine, haha).

Feel free to fork it (modify, run, or ...)

Basic instructions to use it:

https://github.com/harlowja/pippin#pippin

-Josh

Bailey, Darragh wrote:

You may find the code for pip-compile
https://github.com/nvie/pip-tools/tree/future of interest for this, as I
think they may already have a solution for the deep dependency analysis.


I've started experimenting with it for git-upstream cause GitPython have
a habbit of breaking stuff through a couple of releases now :-(


What I like is:
* Doesn't require an extra tool before using 'pip install'
** Some may want to regen the dependencies, but it's optional and the
common python dev approach is retained
* Stable releases are guaranteed to use the versions of dependencies
they were released and verified against
* Improves on the guarantee of gated branch CI
** The idea that if you sync with upstream any test failures are due to
your local changes
** Which is not always true if updated deps can break stuff


On the flip side:
* You remain exposed to security issues in python code until you
manually update
* Development cycle doesn't move forward automatically, may not see
compatibility issues until late when forced to move forward one of the
deps


Think the cons can be handled by some additional CI jobs to update the
pins on a regular basis and pass it through the standard gates and
potentially to auto approve during development cycles if they pass
(already getting the latest matching ones so no big diff here). Some
decisions on trade off around whether this should be done for stable
releases automatically or periodically requiring manual approval would
have to be made.


Did I say how much I like the fact that it doesn't require another tool
before just being able to use 'pip install'?


To experiment with it:
virtualenv .venv/pip-tools
source .venv/pip-tools/bin/activate
pip install git+https://github.com/nvie/pip-tools.git@future

Regards,
Darragh Bailey

Nothing is foolproof to a sufficiently talented fool - Unknown

On 22/01/15 03:45, Joshua Harlow wrote:

A slightly better version that starts to go deeper (and downloads
dependencies of dependencies and extracts there egg_info to get at
these dependencies...)

https://gist.github.com/harlowja/555ea019aef4e901897b

Output @ http://paste.ubuntu.com/9813919/

When ran on the same 'test.txt' mentioned below...

Happy hacking!

-Josh

Joshua Harlow wrote:

A run that shows more of the happy/desired path:

$ cat test.txt
six1
taskflow0.5
$ python pippin.py -r test.txt
Initial package set:
- six ['1']
- taskflow ['0.5']
Deep package set:
- six ['==1.9.0']
- taskflow ['==0.4.0']

-Josh

Joshua Harlow wrote:

Another thing that I just started whipping together:

https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f

The idea for the above is to use pip to download dependencies, but
figure out what versions will work using our own resolver (and our own
querying of 'http://pypi.python.org/pypi/%s/json') that just does a
very
deep search of all requirements (and requirements of requirements...).

The idea for that is that the probe() function in that gist will
'freeze' a single requirement then dive down into further requirements
and ensure compatibility while that 'diving' (aka, recursion into
further requirements) is underway. If a incompatibility is found then
the recursion will back-track and try a to freeze a different
version of
a desired package (and repeat...).

To me this kind of deep finding would be a potential way of making
this
work in a way that basically only uses pip for downloading (and does
the
deep matching/probing) on our own since once the algorithm above
doesn't
backtrack and finds a matching set of requirements that will all work
together the program can exit (and this set can then be used as the
master set for openstack; at that point we might have to tell
people to
not use pip, or to only use pip --download to fetch the compatible
versions).

It's not completed but it could be complementary to what others are
working on; feel free to hack away :)

So far the following works:

$ cat test.txt
six1
taskflow1

$ python pippin.py -r test.txt
Initial package set:
- six ['1']
- taskflow ['1']
Traceback (most recent call last):
File pippin.py, line 168, inmodule
main()
File pippin.py, line 162, in main
matches = probe(initial, {})
File pippin.py, line 139, in probe
result = probe(requirements, 

Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-30 Thread Joshua Harlow

Cool,

I've got to try that out today to see what it's doing.

I've also shoved my little program up @ 
https://github.com/harlowja/pippin (the pip-tools one is definitely more 
elegantly coded than mine, haha).


Feel free to fork it (modify, run, or ...)

Basic instructions to use it:

https://github.com/harlowja/pippin#pippin

-Josh

Bailey, Darragh wrote:

You may find the code for pip-compile
https://github.com/nvie/pip-tools/tree/future of interest for this, as I
think they may already have a solution for the deep dependency analysis.


I've started experimenting with it for git-upstream cause GitPython have
a habbit of breaking stuff through a couple of releases now :-(


What I like is:
* Doesn't require an extra tool before using 'pip install'
** Some may want to regen the dependencies, but it's optional and the
common python dev approach is retained
* Stable releases are guaranteed to use the versions of dependencies
they were released and verified against
* Improves on the guarantee of gated branch CI
** The idea that if you sync with upstream any test failures are due to
your local changes
** Which is not always true if updated deps can break stuff


On the flip side:
* You remain exposed to security issues in python code until you
manually update
* Development cycle doesn't move forward automatically, may not see
compatibility issues until late when forced to move forward one of the deps


Think the cons can be handled by some additional CI jobs to update the
pins on a regular basis and pass it through the standard gates and
potentially to auto approve during development cycles if they pass
(already getting the latest matching ones so no big diff here). Some
decisions on trade off around whether this should be done for stable
releases automatically or periodically requiring manual approval would
have to be made.


Did I say how much I like the fact that it doesn't require another tool
before just being able to use 'pip install'?


To experiment with it:
virtualenv .venv/pip-tools
source .venv/pip-tools/bin/activate
pip install git+https://github.com/nvie/pip-tools.git@future

Regards,
Darragh Bailey

Nothing is foolproof to a sufficiently talented fool - Unknown

On 22/01/15 03:45, Joshua Harlow wrote:

A slightly better version that starts to go deeper (and downloads
dependencies of dependencies and extracts there egg_info to get at
these dependencies...)

https://gist.github.com/harlowja/555ea019aef4e901897b

Output @ http://paste.ubuntu.com/9813919/

When ran on the same 'test.txt' mentioned below...

Happy hacking!

-Josh

Joshua Harlow wrote:

A run that shows more of the happy/desired path:

$ cat test.txt
six1
taskflow0.5
$ python pippin.py -r test.txt
Initial package set:
- six ['1']
- taskflow ['0.5']
Deep package set:
- six ['==1.9.0']
- taskflow ['==0.4.0']

-Josh

Joshua Harlow wrote:

Another thing that I just started whipping together:

https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f

The idea for the above is to use pip to download dependencies, but
figure out what versions will work using our own resolver (and our own
querying of 'http://pypi.python.org/pypi/%s/json') that just does a
very
deep search of all requirements (and requirements of requirements...).

The idea for that is that the probe() function in that gist will
'freeze' a single requirement then dive down into further requirements
and ensure compatibility while that 'diving' (aka, recursion into
further requirements) is underway. If a incompatibility is found then
the recursion will back-track and try a to freeze a different
version of
a desired package (and repeat...).

To me this kind of deep finding would be a potential way of making this
work in a way that basically only uses pip for downloading (and does
the
deep matching/probing) on our own since once the algorithm above
doesn't
backtrack and finds a matching set of requirements that will all work
together the program can exit (and this set can then be used as the
master set for openstack; at that point we might have to tell people to
not use pip, or to only use pip --download to fetch the compatible
versions).

It's not completed but it could be complementary to what others are
working on; feel free to hack away :)

So far the following works:

$ cat test.txt
six1
taskflow1

$ python pippin.py -r test.txt
Initial package set:
- six ['1']
- taskflow ['1']
Traceback (most recent call last):
File pippin.py, line 168, inmodule
main()
File pippin.py, line 162, in main
matches = probe(initial, {})
File pippin.py, line 139, in probe
result = probe(requirements, gathered)
File pippin.py, line 129, in probe
m = find_match(pkg_name, req)
File pippin.py, line 112, in find_match
return match_available(req.req, find_versions(pkg_name))
File pippin.py, line 108, in match_available
 matches '%s' (tried %s) % (req, looked_in))
__main__.NotFoundException: No requirement found that matches
'taskflow1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
'0.2', 

Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-30 Thread Bailey, Darragh

You may find the code for pip-compile
https://github.com/nvie/pip-tools/tree/future of interest for this, as I
think they may already have a solution for the deep dependency analysis.


I've started experimenting with it for git-upstream cause GitPython have
a habbit of breaking stuff through a couple of releases now :-(


What I like is:
* Doesn't require an extra tool before using 'pip install'
** Some may want to regen the dependencies, but it's optional and the
common python dev approach is retained
* Stable releases are guaranteed to use the versions of dependencies
they were released and verified against
* Improves on the guarantee of gated branch CI
** The idea that if you sync with upstream any test failures are due to
your local changes
** Which is not always true if updated deps can break stuff


On the flip side:
* You remain exposed to security issues in python code until you
manually update
* Development cycle doesn't move forward automatically, may not see
compatibility issues until late when forced to move forward one of the deps


Think the cons can be handled by some additional CI jobs to update the
pins on a regular basis and pass it through the standard gates and
potentially to auto approve during development cycles if they pass
(already getting the latest matching ones so no big diff here). Some
decisions on trade off around whether this should be done for stable
releases automatically or periodically requiring manual approval would
have to be made.


Did I say how much I like the fact that it doesn't require another tool
before just being able to use 'pip install'?


To experiment with it:
virtualenv .venv/pip-tools
source .venv/pip-tools/bin/activate
pip install git+https://github.com/nvie/pip-tools.git@future

Regards,
Darragh Bailey

Nothing is foolproof to a sufficiently talented fool - Unknown

On 22/01/15 03:45, Joshua Harlow wrote:
 A slightly better version that starts to go deeper (and downloads
 dependencies of dependencies and extracts there egg_info to get at
 these dependencies...)

 https://gist.github.com/harlowja/555ea019aef4e901897b

 Output @ http://paste.ubuntu.com/9813919/

 When ran on the same 'test.txt' mentioned below...

 Happy hacking!

 -Josh

 Joshua Harlow wrote:
 A run that shows more of the happy/desired path:

 $ cat test.txt
 six1
 taskflow0.5
 $ python pippin.py -r test.txt
 Initial package set:
 - six ['1']
 - taskflow ['0.5']
 Deep package set:
 - six ['==1.9.0']
 - taskflow ['==0.4.0']

 -Josh

 Joshua Harlow wrote:
 Another thing that I just started whipping together:

 https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f

 The idea for the above is to use pip to download dependencies, but
 figure out what versions will work using our own resolver (and our own
 querying of 'http://pypi.python.org/pypi/%s/json') that just does a
 very
 deep search of all requirements (and requirements of requirements...).

 The idea for that is that the probe() function in that gist will
 'freeze' a single requirement then dive down into further requirements
 and ensure compatibility while that 'diving' (aka, recursion into
 further requirements) is underway. If a incompatibility is found then
 the recursion will back-track and try a to freeze a different
 version of
 a desired package (and repeat...).

 To me this kind of deep finding would be a potential way of making this
 work in a way that basically only uses pip for downloading (and does
 the
 deep matching/probing) on our own since once the algorithm above
 doesn't
 backtrack and finds a matching set of requirements that will all work
 together the program can exit (and this set can then be used as the
 master set for openstack; at that point we might have to tell people to
 not use pip, or to only use pip --download to fetch the compatible
 versions).

 It's not completed but it could be complementary to what others are
 working on; feel free to hack away :)

 So far the following works:

 $ cat test.txt
 six1
 taskflow1

 $ python pippin.py -r test.txt
 Initial package set:
 - six ['1']
 - taskflow ['1']
 Traceback (most recent call last):
 File pippin.py, line 168, in module
 main()
 File pippin.py, line 162, in main
 matches = probe(initial, {})
 File pippin.py, line 139, in probe
 result = probe(requirements, gathered)
 File pippin.py, line 129, in probe
 m = find_match(pkg_name, req)
 File pippin.py, line 112, in find_match
 return match_available(req.req, find_versions(pkg_name))
 File pippin.py, line 108, in match_available
  matches '%s' (tried %s) % (req, looked_in))
 __main__.NotFoundException: No requirement found that matches
 'taskflow1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
 '0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])

 I suspect all that is needed to add is the code that is marked with
 FIXME/TODO there and this kind of recursive back-tracking might just do
 the trick...

 -Josh

 Joe Gordon wrote:


 On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon joe.gord...@gmail.com
 

Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-27 Thread Joshua Harlow

Here is one that is more complete:

https://gist.github.com/harlowja/e22e8c0771b336ca392f

Using a really simple requirement set that will not work:

$ cat test.txt
taskflow
networkx1.5

Running the above gist on that (and waiting for a while; since it does 
do a lot of backtracking and trying of different versions) the 
program will eventually break with something like the following:


http://paste.ubuntu.com/9907530/

The above can likely be optimized more (to avoid diving into 
requirements that have already failed being selected) but it does seem 
to work as a satisfiability solver that will recursively check 
requirements deeply and try as many solutions as possible (with 
backtracking when a requirement is selected that doesn't work).


-Josh

Joshua Harlow wrote:

Seems like a simple fix?

https://github.com/pypa/pip/blob/1.5.6/pip/req.py#L1536

Make a new session somewhere in that gist/code and profit?

Kevin L. Mitchell wrote:

On Wed, 2015-01-21 at 18:48 -0800, Joshua Harlow wrote:

Another thing that I just started whipping together:

https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f


One problem, though, is that parse_requirements() now requires the
session keyword argument. In version 6.0.6, parse_requirements() begins
with:

def parse_requirements(filename, finder=None, comes_from=None,
options=None,
session=None):
if session is None:
raise TypeError(
parse_requirements() missing 1 required keyword argument: 
'session'
)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-27 Thread Joshua Harlow
Btw try this on 'pip6' (for those that want to); Pip 6+ moved some of 
the code around that this uses; for feel to update it though and adjust 
it to find where the new stuff is :-P


Joshua Harlow wrote:

Here is one that is more complete:

https://gist.github.com/harlowja/e22e8c0771b336ca392f

Using a really simple requirement set that will not work:

$ cat test.txt
taskflow
networkx1.5

Running the above gist on that (and waiting for a while; since it does
do a lot of backtracking and trying of different versions) the
program will eventually break with something like the following:

http://paste.ubuntu.com/9907530/

The above can likely be optimized more (to avoid diving into
requirements that have already failed being selected) but it does seem
to work as a satisfiability solver that will recursively check
requirements deeply and try as many solutions as possible (with
backtracking when a requirement is selected that doesn't work).

-Josh

Joshua Harlow wrote:

Seems like a simple fix?

https://github.com/pypa/pip/blob/1.5.6/pip/req.py#L1536

Make a new session somewhere in that gist/code and profit?

Kevin L. Mitchell wrote:

On Wed, 2015-01-21 at 18:48 -0800, Joshua Harlow wrote:

Another thing that I just started whipping together:

https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f


One problem, though, is that parse_requirements() now requires the
session keyword argument. In version 6.0.6, parse_requirements() begins
with:

def parse_requirements(filename, finder=None, comes_from=None,
options=None,
session=None):
if session is None:
raise TypeError(
parse_requirements() missing 1 required keyword argument: 
'session'
)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-27 Thread Matt Riedemann



On 1/15/2015 5:41 PM, Sean Dague wrote:

On 01/15/2015 06:25 PM, Joe Gordon wrote:

We can side step the dependency graphing and ordering issue by looking
at the list of curently installed packages via pip freeze and not
installing dependencies (pip install --no-deps)

After looking into this further here are the known issues:

* Partial capping won't work [0], so we need to pin all dependencies, we
can generate this list per file via pip install -r and pip freeze,
but this doesn't address the issue of apt-get vs pip install. For
example in the stable gate we use suds 0.4.1 but only suds 0.4.0 is
available via pip.
* Not all packages are installed in are standard dsvm-tempest env, so
using pip-freeze from that isn't enough
* We need to run this per requirements file and move to using pip
install --no-deps everywhere. As the global-requirements sync wouldn't
work the first time since files don't list all transient dependencies yet.


So that's basically writing our own dependency system entirely, and
skipping pip's (vs. fudging around pip's issues). I expect that's going
to go poorly in the OpenStack ecosystem realm because a lot of very
repetitive manual analysis will need to be done on each project. And if
we want to bump a cap, regenerating all the graphs becomes another
manual process.


* We can still break if a package version is removed from pypi
* in pip-freeze we sometimes install versions lower then our minimum
version (python-libvirt!)


That's because python-libvirt is not in any requirements.txt file, so
we're taking the system version.


It's in test-requirements.txt, but not for long. [1]



-Sean



[1] https://review.openstack.org/#/c/150148/

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-22 Thread Kevin L. Mitchell
On Wed, 2015-01-21 at 18:48 -0800, Joshua Harlow wrote:
 Another thing that I just started whipping together:
 
 https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f

One problem, though, is that parse_requirements() now requires the
session keyword argument.  In version 6.0.6, parse_requirements() begins
with:

def parse_requirements(filename, finder=None, comes_from=None, 
options=None,
   session=None):
if session is None:
raise TypeError(
parse_requirements() missing 1 required keyword argument: 
'session'
)
-- 
Kevin L. Mitchell kevin.mitch...@rackspace.com
Rackspace


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-22 Thread Joshua Harlow

Seems like a simple fix?

https://github.com/pypa/pip/blob/1.5.6/pip/req.py#L1536

Make a new session somewhere in that gist/code and profit?

Kevin L. Mitchell wrote:

On Wed, 2015-01-21 at 18:48 -0800, Joshua Harlow wrote:

Another thing that I just started whipping together:

https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f


One problem, though, is that parse_requirements() now requires the
session keyword argument.  In version 6.0.6, parse_requirements() begins
with:

 def parse_requirements(filename, finder=None, comes_from=None, 
options=None,
session=None):
 if session is None:
 raise TypeError(
 parse_requirements() missing 1 required keyword argument: 

 'session'
 )


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-21 Thread Joshua Harlow

Another thing that I just started whipping together:

https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f

The idea for the above is to use pip to download dependencies, but 
figure out what versions will work using our own resolver (and our own 
querying of 'http://pypi.python.org/pypi/%s/json') that just does a very 
deep search of all requirements (and requirements of requirements...).


The idea for that is that the probe() function in that gist will 
'freeze' a single requirement then dive down into further requirements 
and ensure compatibility while that 'diving' (aka, recursion into 
further requirements) is underway. If a incompatibility is found then 
the recursion will back-track and try a to freeze a different version of 
a desired package (and repeat...).


To me this kind of deep finding would be a potential way of making this 
work in a way that basically only uses pip for downloading (and does the 
deep matching/probing) on our own since once the algorithm above doesn't 
backtrack and finds a matching set of requirements that will all work 
together the program can exit (and this set can then be used as the 
master set for openstack; at that point we might have to tell people to 
not use pip, or to only use pip --download to fetch the compatible 
versions).


It's not completed but it could be complementary to what others are 
working on; feel free to hack away :)


So far the following works:

$ cat test.txt
six1
taskflow1

$ python pippin.py  -r test.txt
Initial package set:
- six ['1']
- taskflow ['1']
Traceback (most recent call last):
  File pippin.py, line 168, in module
main()
  File pippin.py, line 162, in main
matches = probe(initial, {})
  File pippin.py, line 139, in probe
result = probe(requirements, gathered)
  File pippin.py, line 129, in probe
m = find_match(pkg_name, req)
  File pippin.py, line 112, in find_match
return match_available(req.req, find_versions(pkg_name))
  File pippin.py, line 108, in match_available
 matches '%s' (tried %s) % (req, looked_in))
__main__.NotFoundException: No requirement found that matches 
'taskflow1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21', 
'0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])


I suspect all that is needed to add is the code that is marked with 
FIXME/TODO there and this kind of recursive back-tracking might just do 
the trick...


-Josh

Joe Gordon wrote:



On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon joe.gord...@gmail.com
mailto:joe.gord...@gmail.com wrote:

We can side step the dependency graphing and ordering issue by
looking at the list of curently installed packages via pip freeze
and not installing dependencies (pip install --no-deps)

After looking into this further here are the known issues:

* Partial capping won't work [0], so we need to pin all
dependencies, we can generate this list per file via pip install
-r and pip freeze, but this doesn't address the issue of apt-get
vs pip install. For example in the stable gate we use suds 0.4.1 but
only suds 0.4.0 is available via pip.
* Not all packages are installed in are standard dsvm-tempest env,
so using pip-freeze from that isn't enough
* We need to run this per requirements file and move to using pip
install --no-deps everywhere. As the global-requirements sync
wouldn't work the first time since files don't list all transient
dependencies yet.
* We can still break if a package version is removed from pypi
* in pip-freeze we sometimes install versions lower then our minimum
version (python-libvirt!)


Exploring a few ideas here: https://review.openstack.org/#/c/147451/4



[0]
http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html

On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley fu...@yuggoth.org
mailto:fu...@yuggoth.org wrote:

On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
[...]
  The other thing that happened was partial capping doesn't work,
  because something else moves forward and breaks you from below. So
  the patch will need to hit everything at once.

Right, and we _have_ to start using stable branches on all
clients/libraries to backport fixes as part of that. This means that
the stable branch management workflow is about to become pervasive
across some teams who were previously (blissfully?) ignorant of it.

  Unresolved entirely is the tertiary dependencies (not direct
  dependencies of any OpenStack project). That will need another
  mechanism to seed them before any installation happens.
[...]

I won't go so far as to call it intractable, but I took a stab at it
about a year ago and building the dependency graph properly to be
able to do a depth-first ordering is nontrivial (enough that after
about a week hacking on possible solutions I gave up and switched to

Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-21 Thread Joshua Harlow

A run that shows more of the happy/desired path:

$ cat test.txt
six1
taskflow0.5
$ python pippin.py  -r test.txt
Initial package set:
- six ['1']
- taskflow ['0.5']
Deep package set:
- six ['==1.9.0']
- taskflow ['==0.4.0']

-Josh

Joshua Harlow wrote:

Another thing that I just started whipping together:

https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f

The idea for the above is to use pip to download dependencies, but
figure out what versions will work using our own resolver (and our own
querying of 'http://pypi.python.org/pypi/%s/json') that just does a very
deep search of all requirements (and requirements of requirements...).

The idea for that is that the probe() function in that gist will
'freeze' a single requirement then dive down into further requirements
and ensure compatibility while that 'diving' (aka, recursion into
further requirements) is underway. If a incompatibility is found then
the recursion will back-track and try a to freeze a different version of
a desired package (and repeat...).

To me this kind of deep finding would be a potential way of making this
work in a way that basically only uses pip for downloading (and does the
deep matching/probing) on our own since once the algorithm above doesn't
backtrack and finds a matching set of requirements that will all work
together the program can exit (and this set can then be used as the
master set for openstack; at that point we might have to tell people to
not use pip, or to only use pip --download to fetch the compatible
versions).

It's not completed but it could be complementary to what others are
working on; feel free to hack away :)

So far the following works:

$ cat test.txt
six1
taskflow1

$ python pippin.py -r test.txt
Initial package set:
- six ['1']
- taskflow ['1']
Traceback (most recent call last):
File pippin.py, line 168, in module
main()
File pippin.py, line 162, in main
matches = probe(initial, {})
File pippin.py, line 139, in probe
result = probe(requirements, gathered)
File pippin.py, line 129, in probe
m = find_match(pkg_name, req)
File pippin.py, line 112, in find_match
return match_available(req.req, find_versions(pkg_name))
File pippin.py, line 108, in match_available
 matches '%s' (tried %s) % (req, looked_in))
__main__.NotFoundException: No requirement found that matches
'taskflow1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
'0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])

I suspect all that is needed to add is the code that is marked with
FIXME/TODO there and this kind of recursive back-tracking might just do
the trick...

-Josh

Joe Gordon wrote:



On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon joe.gord...@gmail.com
mailto:joe.gord...@gmail.com wrote:

We can side step the dependency graphing and ordering issue by
looking at the list of curently installed packages via pip freeze
and not installing dependencies (pip install --no-deps)

After looking into this further here are the known issues:

* Partial capping won't work [0], so we need to pin all
dependencies, we can generate this list per file via pip install
-r and pip freeze, but this doesn't address the issue of apt-get
vs pip install. For example in the stable gate we use suds 0.4.1 but
only suds 0.4.0 is available via pip.
* Not all packages are installed in are standard dsvm-tempest env,
so using pip-freeze from that isn't enough
* We need to run this per requirements file and move to using pip
install --no-deps everywhere. As the global-requirements sync
wouldn't work the first time since files don't list all transient
dependencies yet.
* We can still break if a package version is removed from pypi
* in pip-freeze we sometimes install versions lower then our minimum
version (python-libvirt!)


Exploring a few ideas here: https://review.openstack.org/#/c/147451/4



[0]
http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html


On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley fu...@yuggoth.org
mailto:fu...@yuggoth.org wrote:

On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
[...]
 The other thing that happened was partial capping doesn't work,
 because something else moves forward and breaks you from below. So
 the patch will need to hit everything at once.

Right, and we _have_ to start using stable branches on all
clients/libraries to backport fixes as part of that. This means that
the stable branch management workflow is about to become pervasive
across some teams who were previously (blissfully?) ignorant of it.

 Unresolved entirely is the tertiary dependencies (not direct
 dependencies of any OpenStack project). That will need another
 mechanism to seed them before any installation happens.
[...]

I won't go so far as to call it intractable, but I took a stab at it
about a year ago and building the dependency graph properly to be
able to do a depth-first ordering is nontrivial (enough that after
about a week hacking on possible solutions I gave up and switched to
more productive tasks). The primary 

Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-21 Thread Joshua Harlow
A slightly better version that starts to go deeper (and downloads 
dependencies of dependencies and extracts there egg_info to get at these 
dependencies...)


https://gist.github.com/harlowja/555ea019aef4e901897b

Output @ http://paste.ubuntu.com/9813919/

When ran on the same 'test.txt' mentioned below...

Happy hacking!

-Josh

Joshua Harlow wrote:

A run that shows more of the happy/desired path:

$ cat test.txt
six1
taskflow0.5
$ python pippin.py -r test.txt
Initial package set:
- six ['1']
- taskflow ['0.5']
Deep package set:
- six ['==1.9.0']
- taskflow ['==0.4.0']

-Josh

Joshua Harlow wrote:

Another thing that I just started whipping together:

https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f

The idea for the above is to use pip to download dependencies, but
figure out what versions will work using our own resolver (and our own
querying of 'http://pypi.python.org/pypi/%s/json') that just does a very
deep search of all requirements (and requirements of requirements...).

The idea for that is that the probe() function in that gist will
'freeze' a single requirement then dive down into further requirements
and ensure compatibility while that 'diving' (aka, recursion into
further requirements) is underway. If a incompatibility is found then
the recursion will back-track and try a to freeze a different version of
a desired package (and repeat...).

To me this kind of deep finding would be a potential way of making this
work in a way that basically only uses pip for downloading (and does the
deep matching/probing) on our own since once the algorithm above doesn't
backtrack and finds a matching set of requirements that will all work
together the program can exit (and this set can then be used as the
master set for openstack; at that point we might have to tell people to
not use pip, or to only use pip --download to fetch the compatible
versions).

It's not completed but it could be complementary to what others are
working on; feel free to hack away :)

So far the following works:

$ cat test.txt
six1
taskflow1

$ python pippin.py -r test.txt
Initial package set:
- six ['1']
- taskflow ['1']
Traceback (most recent call last):
File pippin.py, line 168, in module
main()
File pippin.py, line 162, in main
matches = probe(initial, {})
File pippin.py, line 139, in probe
result = probe(requirements, gathered)
File pippin.py, line 129, in probe
m = find_match(pkg_name, req)
File pippin.py, line 112, in find_match
return match_available(req.req, find_versions(pkg_name))
File pippin.py, line 108, in match_available
 matches '%s' (tried %s) % (req, looked_in))
__main__.NotFoundException: No requirement found that matches
'taskflow1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
'0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])

I suspect all that is needed to add is the code that is marked with
FIXME/TODO there and this kind of recursive back-tracking might just do
the trick...

-Josh

Joe Gordon wrote:



On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon joe.gord...@gmail.com
mailto:joe.gord...@gmail.com wrote:

We can side step the dependency graphing and ordering issue by
looking at the list of curently installed packages via pip freeze
and not installing dependencies (pip install --no-deps)

After looking into this further here are the known issues:

* Partial capping won't work [0], so we need to pin all
dependencies, we can generate this list per file via pip install
-r and pip freeze, but this doesn't address the issue of apt-get
vs pip install. For example in the stable gate we use suds 0.4.1 but
only suds 0.4.0 is available via pip.
* Not all packages are installed in are standard dsvm-tempest env,
so using pip-freeze from that isn't enough
* We need to run this per requirements file and move to using pip
install --no-deps everywhere. As the global-requirements sync
wouldn't work the first time since files don't list all transient
dependencies yet.
* We can still break if a package version is removed from pypi
* in pip-freeze we sometimes install versions lower then our minimum
version (python-libvirt!)


Exploring a few ideas here: https://review.openstack.org/#/c/147451/4



[0]
http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html



On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley fu...@yuggoth.org
mailto:fu...@yuggoth.org wrote:

On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
[...]
 The other thing that happened was partial capping doesn't work,
 because something else moves forward and breaks you from below. So
 the patch will need to hit everything at once.

Right, and we _have_ to start using stable branches on all
clients/libraries to backport fixes as part of that. This means that
the stable branch management workflow is about to become pervasive
across some teams who were previously (blissfully?) ignorant of it.

 Unresolved entirely is the tertiary dependencies (not direct
 dependencies of any OpenStack project). That will need another
 mechanism to seed them 

Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-16 Thread Bailey, Darragh

On 15/01/15 23:41, Sean Dague wrote:
 On 01/15/2015 06:25 PM, Joe Gordon wrote:
 We can side step the dependency graphing and ordering issue by looking
 at the list of curently installed packages via pip freeze and not
 installing dependencies (pip install --no-deps)

 After looking into this further here are the known issues:

 * Partial capping won't work [0], so we need to pin all dependencies, we
 can generate this list per file via pip install -r and pip freeze,
 but this doesn't address the issue of apt-get vs pip install. For
 example in the stable gate we use suds 0.4.1 but only suds 0.4.0 is
 available via pip.
 * Not all packages are installed in are standard dsvm-tempest env, so
 using pip-freeze from that isn't enough
 * We need to run this per requirements file and move to using pip
 install --no-deps everywhere. As the global-requirements sync wouldn't
 work the first time since files don't list all transient dependencies yet.
 So that's basically writing our own dependency system entirely, and
 skipping pip's (vs. fudging around pip's issues). I expect that's going
 to go poorly in the OpenStack ecosystem realm because a lot of very
 repetitive manual analysis will need to be done on each project. And if
 we want to bump a cap, regenerating all the graphs becomes another
 manual process.

Pip's loose dependency processing where each entry is treated as an
entirely separate resolution with no consideration of previous or later
constraints came up recently in a discussion at work.

Looking around I discovered someone had already started working on a
solution, which involves moving the current requirements to a
'requirements.in' file and then compiling that to requirements.txt,
which would contain only fully pinned packages in the correct order for
pip to install them without needing to add any special options.

Might be worth seeing if that could be developed further:
http://nvie.com/posts/better-package-management/
https://github.com/nvie/pip-tools/tree/future



 * We can still break if a package version is removed from pypi
 * in pip-freeze we sometimes install versions lower then our minimum
 version (python-libvirt!)
 That's because python-libvirt is not in any requirements.txt file, so
 we're taking the system version.

   -Sean


Regards,
Darragh Bailey

Nothing is foolproof to a sufficiently talented fool - Unknown


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-15 Thread Jeremy Stanley
On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
[...]
 The other thing that happened was partial capping doesn't work,
 because something else moves forward and breaks you from below. So
 the patch will need to hit everything at once.

Right, and we _have_ to start using stable branches on all
clients/libraries to backport fixes as part of that. This means that
the stable branch management workflow is about to become pervasive
across some teams who were previously (blissfully?) ignorant of it.

 Unresolved entirely is the tertiary dependencies (not direct
 dependencies of any OpenStack project). That will need another
 mechanism to seed them before any installation happens.
[...]

I won't go so far as to call it intractable, but I took a stab at it
about a year ago and building the dependency graph properly to be
able to do a depth-first ordering is nontrivial (enough that after
about a week hacking on possible solutions I gave up and switched to
more productive tasks). The primary complications I ran into were
identifying setup_requires in transitive dependencies and dealing
with platform/version-specific dependencies. That said, there's a
very good chance that more recent improvements in setuptools, pip
and virtualenv could make this task easier.

 That's the things I can think off from the top of my head.

The implementation, from a devstack-gate perspective, is also going
to require a decision on whether we stick with stable/relname for
branches of libraries too or switch to some extended branch mapping
mechanism to be able to track stable/relnum branches for those. And
we're going to need more jobs to ensure that clients (specifically)
retain backward-compatibility from an appdev and end user
perspective since they'll no longer get any testing as server
dependencies on stable branches (due to being capped there).
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-15 Thread Joe Gordon
On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon joe.gord...@gmail.com wrote:

 We can side step the dependency graphing and ordering issue by looking at
 the list of curently installed packages via pip freeze and not installing
 dependencies (pip install --no-deps)

 After looking into this further here are the known issues:

 * Partial capping won't work [0], so we need to pin all dependencies, we
 can generate this list per file via pip install -r and pip freeze, but
 this doesn't address the issue of apt-get vs pip install. For example in
 the stable gate we use suds 0.4.1 but only suds 0.4.0 is available via pip.
 * Not all packages are installed in are standard dsvm-tempest env, so
 using pip-freeze from that isn't enough
 * We need to run this per requirements file and move to using pip install
 --no-deps everywhere. As the global-requirements sync wouldn't work the
 first time since files don't list all transient dependencies yet.
 * We can still break if a package version is removed from pypi
 * in pip-freeze we sometimes install versions lower then our minimum
 version (python-libvirt!)


Exploring a few ideas here: https://review.openstack.org/#/c/147451/4




 [0]
 http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html

 On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley fu...@yuggoth.org wrote:

 On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
 [...]
  The other thing that happened was partial capping doesn't work,
  because something else moves forward and breaks you from below. So
  the patch will need to hit everything at once.

 Right, and we _have_ to start using stable branches on all
 clients/libraries to backport fixes as part of that. This means that
 the stable branch management workflow is about to become pervasive
 across some teams who were previously (blissfully?) ignorant of it.

  Unresolved entirely is the tertiary dependencies (not direct
  dependencies of any OpenStack project). That will need another
  mechanism to seed them before any installation happens.
 [...]

 I won't go so far as to call it intractable, but I took a stab at it
 about a year ago and building the dependency graph properly to be
 able to do a depth-first ordering is nontrivial (enough that after
 about a week hacking on possible solutions I gave up and switched to
 more productive tasks). The primary complications I ran into were
 identifying setup_requires in transitive dependencies and dealing
 with platform/version-specific dependencies. That said, there's a
 very good chance that more recent improvements in setuptools, pip
 and virtualenv could make this task easier.

  That's the things I can think off from the top of my head.

 The implementation, from a devstack-gate perspective, is also going
 to require a decision on whether we stick with stable/relname for
 branches of libraries too or switch to some extended branch mapping
 mechanism to be able to track stable/relnum branches for those. And
 we're going to need more jobs to ensure that clients (specifically)
 retain backward-compatibility from an appdev and end user
 perspective since they'll no longer get any testing as server
 dependencies on stable branches (due to being capped there).
 --
 Jeremy Stanley

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-15 Thread Sean Dague
On 01/15/2015 06:25 PM, Joe Gordon wrote:
 We can side step the dependency graphing and ordering issue by looking
 at the list of curently installed packages via pip freeze and not
 installing dependencies (pip install --no-deps)
 
 After looking into this further here are the known issues:
 
 * Partial capping won't work [0], so we need to pin all dependencies, we
 can generate this list per file via pip install -r and pip freeze,
 but this doesn't address the issue of apt-get vs pip install. For
 example in the stable gate we use suds 0.4.1 but only suds 0.4.0 is
 available via pip.
 * Not all packages are installed in are standard dsvm-tempest env, so
 using pip-freeze from that isn't enough
 * We need to run this per requirements file and move to using pip
 install --no-deps everywhere. As the global-requirements sync wouldn't
 work the first time since files don't list all transient dependencies yet.

So that's basically writing our own dependency system entirely, and
skipping pip's (vs. fudging around pip's issues). I expect that's going
to go poorly in the OpenStack ecosystem realm because a lot of very
repetitive manual analysis will need to be done on each project. And if
we want to bump a cap, regenerating all the graphs becomes another
manual process.

 * We can still break if a package version is removed from pypi
 * in pip-freeze we sometimes install versions lower then our minimum
 version (python-libvirt!)

That's because python-libvirt is not in any requirements.txt file, so
we're taking the system version.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-15 Thread Joe Gordon
We can side step the dependency graphing and ordering issue by looking at
the list of curently installed packages via pip freeze and not installing
dependencies (pip install --no-deps)

After looking into this further here are the known issues:

* Partial capping won't work [0], so we need to pin all dependencies, we
can generate this list per file via pip install -r and pip freeze, but
this doesn't address the issue of apt-get vs pip install. For example in
the stable gate we use suds 0.4.1 but only suds 0.4.0 is available via pip.
* Not all packages are installed in are standard dsvm-tempest env, so using
pip-freeze from that isn't enough
* We need to run this per requirements file and move to using pip install
--no-deps everywhere. As the global-requirements sync wouldn't work the
first time since files don't list all transient dependencies yet.
* We can still break if a package version is removed from pypi
* in pip-freeze we sometimes install versions lower then our minimum
version (python-libvirt!)



[0]
http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html

On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley fu...@yuggoth.org wrote:

 On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
 [...]
  The other thing that happened was partial capping doesn't work,
  because something else moves forward and breaks you from below. So
  the patch will need to hit everything at once.

 Right, and we _have_ to start using stable branches on all
 clients/libraries to backport fixes as part of that. This means that
 the stable branch management workflow is about to become pervasive
 across some teams who were previously (blissfully?) ignorant of it.

  Unresolved entirely is the tertiary dependencies (not direct
  dependencies of any OpenStack project). That will need another
  mechanism to seed them before any installation happens.
 [...]

 I won't go so far as to call it intractable, but I took a stab at it
 about a year ago and building the dependency graph properly to be
 able to do a depth-first ordering is nontrivial (enough that after
 about a week hacking on possible solutions I gave up and switched to
 more productive tasks). The primary complications I ran into were
 identifying setup_requires in transitive dependencies and dealing
 with platform/version-specific dependencies. That said, there's a
 very good chance that more recent improvements in setuptools, pip
 and virtualenv could make this task easier.

  That's the things I can think off from the top of my head.

 The implementation, from a devstack-gate perspective, is also going
 to require a decision on whether we stick with stable/relname for
 branches of libraries too or switch to some extended branch mapping
 mechanism to be able to track stable/relnum branches for those. And
 we're going to need more jobs to ensure that clients (specifically)
 retain backward-compatibility from an appdev and end user
 perspective since they'll no longer get any testing as server
 dependencies on stable branches (due to being capped there).
 --
 Jeremy Stanley

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-15 Thread ZhiQiang Fan
+1

how about the deep dependency? for example, we depends package A, and pin
it, but A-B-C, then B and C are not pined since they are not directly
depended, then what should we do? pin everything?

On Thu, Jan 15, 2015 at 5:35 PM, Joe Gordon joe.gord...@gmail.com wrote:

 Eventlet released 0.16.1 on 2015-01-14 [0], which removed a deprecated
 API that nova stable/* still used. This caused nova-compute in stable/juno
 and stable/icehouse to crash thus breaking grenade on master. In 24 hours
 this bug caused 671 grenade jobs to fail[1]!

 After some quick debugging of this cryptic error suppressing stacktrace
 [2] we got to the real stacktrace:
  File nova/virt/libvirt/__init__.py, line 15, in module

 from nova.virt.libvirt import driver
   File nova/virt/libvirt/driver.py, line 48, in module
 from eventlet import util as eventlet_util

 ImportError: cannot import name util [3]
 and capped the eventlet version on stable/* [4]

 As dependency bugs go this was a pretty typical one, so why am I writing
 this? Because we knew about this bug before it hit the gate, yet it was
 still an issue. The util module was removed in 0.16.0 but 'sneaked into'
 the build [5] so 0.16.1 fixed that. Before 0.16.1 was released this bug
 only impacted people who installed eventlet 0.16.0 from source and not from
 pip. Luckily someone did this and filed a bug [6] and a fix for this was
 landed on master on January 7th, and by the 10th the fix was backported to
 stable/juno and ready [7].  In the mean time, we had an unusually bad week
 dealing with new library versions; boto 2.35.0 was released and broke
 master and stable/juno [8] and sqlalchemy-migrate broke glance on master
 and stable as well [9].  And by the time these issues were fixed, and the
 tests would pass, eventlet 0.16.1 was already out. Unfortunately, there are
 a very small number of people who work on fixing dependency issues, some of
 them were traveling and the rest were over worked, so figuring out what
 went wrong with all the new packages came down to a handful of overworked
 people.

 So how could we have avoided this problem? By capping stable branch
 requirements so we only have to worry about uncapped dependencies on
 master. Capping stable branches has been previous discussed but no action
 has been taken. So going forward I propose we pin all requirements,
 including transitive, on stable branches. This way the release of new
 dependencies cannot automatically break stable branches and thus break
 grenade on master.

 I think we can implement this using `pip freeze` to get a list of all the
 installed libraries and `pip install -r --no-deps` to install the specific
 package versions.  I am not sure how best to handle package versions being
 removed from pypi though. Or at the very least we can cap requirements in
 global requirements and at least reduce the number of packages we install
 uncapped versions of.

 Thoughts?

 best,
 Joe

 [0] https://pypi.python.org/pypi/eventlet/0.16.1
 [1] http://status.openstack.org/elastic-recheck/#1410626
 [2]
 http://logs.openstack.org/98/115998/59/check/check-grenade-dsvm/e57eb40/logs/old/screen-n-cpu.txt.gz
 [3] http://paste.openstack.org/show/157558/
 [4]
 https://review.openstack.org/#/q/I4bbbeb5bf9c22ed36f5c9a74fec6b487d2c15697,n,z
 [5] https://github.com/eventlet/eventlet/releases/tag/v0.16.1
 [6] https://bugs.launchpad.net/nova/+bug/1407685
 [7]
 https://review.openstack.org/#/q/Idbb9d2b53829dae0e807cd1260dee3dce155d5f3,n,z
 [8] https://bugs.launchpad.net/nova/+bug/1408987
 [9] https://bugs.launchpad.net/glance/+bug/1410494

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
blog: zqfan.github.com
git: github.com/zqfan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-15 Thread Nikola ─Éipanov
On 01/15/2015 10:35 AM, Joe Gordon wrote:

 So how could we have avoided this problem? By capping stable branch
 requirements so we only have to worry about uncapped dependencies on
 master. Capping stable branches has been previous discussed but no
 action has been taken. So going forward I propose we pin all
 requirements, including transitive, on stable branches. This way the
 release of new dependencies cannot automatically break stable branches
 and thus break grenade on master.
 

This is an absolute must IMHO, including transitive dependencies,
because if they are not capped - they can cause other issues like bring
in additional deps a stable release is not even supposed to have, among
all the usual issues.

The problem as I understand it is that this breaks how we do upgrades
testing in the gate, AKA the granade job (all in a single VM, install
everything from pip). IMHO this is broken and needs to be fixed ASAP, if
capping breaks it.

N.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-15 Thread Sean Dague
On 01/15/2015 05:21 AM, Nikola ─Éipanov wrote:
 On 01/15/2015 10:35 AM, Joe Gordon wrote:
 
 So how could we have avoided this problem? By capping stable branch
 requirements so we only have to worry about uncapped dependencies on
 master. Capping stable branches has been previous discussed but no
 action has been taken. So going forward I propose we pin all
 requirements, including transitive, on stable branches. This way the
 release of new dependencies cannot automatically break stable branches
 and thus break grenade on master.

 
 This is an absolute must IMHO, including transitive dependencies,
 because if they are not capped - they can cause other issues like bring
 in additional deps a stable release is not even supposed to have, among
 all the usual issues.
 
 The problem as I understand it is that this breaks how we do upgrades
 testing in the gate, AKA the granade job (all in a single VM, install
 everything from pip). IMHO this is broken and needs to be fixed ASAP, if
 capping breaks it.

That's a mis understanding of the last issue, grenade handles this fine.

The capping only caused a problem because we were upgrading components
in an incorrect issue: Swift upgrading before Ceilometer, even though
Ceilometer had middleware installed in Swift. Which meant that for
almost ever we were actually running new Swift with old Ceilometer
middleware.

That was fixed here - https://review.openstack.org/#/c/142075/

The policy decision to cap stable requirements was agreed to previously
(and at summit this year), it just needs someone to implement. This also
came up in this keystoneclient thread from last week -
http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html

Volunteers welcomed.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-15 Thread Sean Dague
On 01/15/2015 08:38 AM, Thierry Carrez wrote:
 Sean Dague wrote:
 The policy decision to cap stable requirements was agreed to previously
 (and at summit this year), it just needs someone to implement.
 
 +1
 
 We started to implement it (in Oslo libraries iirc) and then reverted
 because we encountered problems. We concluded a proper cross-project
 spec should be written so that we can try to anticipate issues better.
 
 I don't have a totally clear view of the issue(s) we encoutered last
 time, though. Could someone write a postmortem or point me to one ? Or
 was the upgrading components issue (fixed in
 https://review.openstack.org/#/c/142075/) the only problem ?

My recollection is the component upgrade order was a big problem, but
not the only one.

The other thing that happened was partial capping doesn't work, because
something else moves forward and breaks you from below. So the patch
will need to hit everything at once.

Unresolved entirely is the tertiary dependencies (not direct
dependencies of any OpenStack project). That will need another mechanism
to seed them before any installation happens.

Whether or not pip should be pinned is also a good question, especially
because pbr requires pip, so the install -U on pbr tends to uplift pip
even if it was previously downgraded.

That's the things I can think off from the top of my head.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate][stable] How eventlet 0.16.1 broke the gate

2015-01-15 Thread Thierry Carrez
Sean Dague wrote:
 The policy decision to cap stable requirements was agreed to previously
 (and at summit this year), it just needs someone to implement.

+1

We started to implement it (in Oslo libraries iirc) and then reverted
because we encountered problems. We concluded a proper cross-project
spec should be written so that we can try to anticipate issues better.

I don't have a totally clear view of the issue(s) we encoutered last
time, though. Could someone write a postmortem or point me to one ? Or
was the upgrading components issue (fixed in
https://review.openstack.org/#/c/142075/) the only problem ?

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev