I've got to try that out today to see what it's doing.

I've also shoved my little program up @ (the pip-tools one is definitely more elegantly coded than mine, haha).

Feel free to fork it (modify, run, or ...)

Basic instructions to use it:


Bailey, Darragh wrote:
You may find the code for pip-compile of interest for this, as I
think they may already have a solution for the deep dependency analysis.

I've started experimenting with it for git-upstream cause GitPython have
a habbit of breaking stuff through a couple of releases now :-(

What I like is:
* Doesn't require an extra tool before using 'pip install'
** Some may want to regen the dependencies, but it's optional and the
common python dev approach is retained
* Stable releases are guaranteed to use the versions of dependencies
they were released and verified against
* Improves on the guarantee of gated branch CI
** The idea that if you sync with upstream any test failures are due to
your local changes
** Which is not always true if updated deps can break stuff

On the flip side:
* You remain exposed to security issues in python code until you
manually update
* Development cycle doesn't move forward automatically, may not see
compatibility issues until late when forced to move forward one of the deps

Think the cons can be handled by some additional CI jobs to update the
pins on a regular basis and pass it through the standard gates and
potentially to auto approve during development cycles if they pass
(already getting the latest matching ones so no big diff here). Some
decisions on trade off around whether this should be done for stable
releases automatically or periodically requiring manual approval would
have to be made.

Did I say how much I like the fact that it doesn't require another tool
before just being able to use 'pip install'?

To experiment with it:
virtualenv .venv/pip-tools
source .venv/pip-tools/bin/activate
pip install git+

Darragh Bailey

"Nothing is foolproof to a sufficiently talented fool" - Unknown

On 22/01/15 03:45, Joshua Harlow wrote:
A slightly better version that starts to go deeper (and downloads
dependencies of dependencies and extracts there egg_info to get at
these dependencies...)

Output @

When ran on the same 'test.txt' mentioned below...

Happy hacking!


Joshua Harlow wrote:
A run that shows more of the happy/desired path:

$ cat test.txt
$ python -r test.txt
Initial package set:
- six ['>1']
- taskflow ['<0.5']
Deep package set:
- six ['==1.9.0']
- taskflow ['==0.4.0']


Joshua Harlow wrote:
Another thing that I just started whipping together:

The idea for the above is to use pip to download dependencies, but
figure out what versions will work using our own resolver (and our own
querying of '') that just does a
deep search of all requirements (and requirements of requirements...).

The idea for that is that the probe() function in that gist will
'freeze' a single requirement then dive down into further requirements
and ensure compatibility while that 'diving' (aka, recursion into
further requirements) is underway. If a incompatibility is found then
the recursion will back-track and try a to freeze a different
version of
a desired package (and repeat...).

To me this kind of deep finding would be a potential way of making this
work in a way that basically only uses pip for downloading (and does
deep matching/probing) on our own since once the algorithm above
backtrack and finds a matching set of requirements that will all work
together the program can exit (and this set can then be used as the
master set for openstack; at that point we might have to tell people to
not use pip, or to only use pip --download to fetch the compatible

It's not completed but it could be complementary to what others are
working on; feel free to hack away :)

So far the following works:

$ cat test.txt

$ python -r test.txt
Initial package set:
- six ['>1']
- taskflow ['>1']
Traceback (most recent call last):
File "", line 168, in<module>
File "", line 162, in main
matches = probe(initial, {})
File "", line 139, in probe
result = probe(requirements, gathered)
File "", line 129, in probe
m = find_match(pkg_name, req)
File "", line 112, in find_match
return match_available(req.req, find_versions(pkg_name))
File "", line 108, in match_available
" matches '%s' (tried %s)" % (req, looked_in))
__main__.NotFoundException: No requirement found that matches
'taskflow>1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
'0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])

I suspect all that is needed to add is the code that is marked with
FIXME/TODO there and this kind of recursive back-tracking might just do
the trick...


Joe Gordon wrote:

On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon<
<>>  wrote:

We can side step the dependency graphing and ordering issue by
looking at the list of curently installed packages via pip freeze
and not installing dependencies (pip install --no-deps)

After looking into this further here are the known issues:

* Partial capping won't work [0], so we need to pin all
dependencies, we can generate this list per file via "pip install
-r" and "pip freeze", but this doesn't address the issue of apt-get
vs pip install. For example in the stable gate we use suds 0.4.1 but
only suds 0.4.0 is available via pip.
* Not all packages are installed in are standard dsvm-tempest env,
so using pip-freeze from that isn't enough
* We need to run this per requirements file and move to using pip
install --no-deps everywhere. As the global-requirements sync
wouldn't work the first time since files don't list all transient
dependencies yet.
* We can still break if a package version is removed from pypi
* in pip-freeze we sometimes install versions lower then our minimum
version (python-libvirt!)

Exploring a few ideas here:


On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley<
<>>  wrote:

On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
The other thing that happened was partial capping doesn't work,
because something else moves forward and breaks you from below. So
the patch will need to hit everything at once.
Right, and we _have_ to start using stable branches on all
clients/libraries to backport fixes as part of that. This means that
the stable branch management workflow is about to become pervasive
across some teams who were previously (blissfully?) ignorant of it.

Unresolved entirely is the tertiary dependencies (not direct
dependencies of any OpenStack project). That will need another
mechanism to seed them before any installation happens.

I won't go so far as to call it intractable, but I took a stab at it
about a year ago and building the dependency graph properly to be
able to do a depth-first ordering is nontrivial (enough that after
about a week hacking on possible solutions I gave up and switched to
more productive tasks). The primary complications I ran into were
identifying setup_requires in transitive dependencies and dealing
with platform/version-specific dependencies. That said, there's a
very good chance that more recent improvements in setuptools, pip
and virtualenv could make this task easier.

That's the things I can think off from the top of my head.
The implementation, from a devstack-gate perspective, is also going
to require a decision on whether we stick with stable/relname for
branches of libraries too or switch to some extended branch mapping
mechanism to be able to track stable/relnum branches for those. And
we're going to need more jobs to ensure that clients (specifically)
retain backward-compatibility from an appdev and end user
perspective since they'll no longer get any testing as server
dependencies on stable branches (due to being capped there).
Jeremy Stanley


OpenStack Development Mailing List (not for usage questions)


OpenStack Development Mailing List (not for usage questions)

OpenStack Development Mailing List (not for usage questions)

OpenStack Development Mailing List (not for usage questions)

OpenStack Development Mailing List (not for usage questions)

OpenStack Development Mailing List (not for usage questions)

Reply via email to