Another thing that I just started whipping together:
https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f
The idea for the above is to use pip to download dependencies, but
figure out what versions will work using our own resolver (and our own
querying of 'http://pypi.python.org/pypi/%s/json') that just does a
very
deep search of all requirements (and requirements of requirements...).
The idea for that is that the probe() function in that gist will
'freeze' a single requirement then dive down into further requirements
and ensure compatibility while that 'diving' (aka, recursion into
further requirements) is underway. If a incompatibility is found then
the recursion will back-track and try a to freeze a different
version of
a desired package (and repeat...).
To me this kind of deep finding would be a potential way of making this
work in a way that basically only uses pip for downloading (and does
the
deep matching/probing) on our own since once the algorithm above
doesn't
backtrack and finds a matching set of requirements that will all work
together the program can exit (and this set can then be used as the
master set for openstack; at that point we might have to tell people to
not use pip, or to only use pip --download to fetch the compatible
versions).
It's not completed but it could be complementary to what others are
working on; feel free to hack away :)
So far the following works:
$ cat test.txt
six>1
taskflow>1
$ python pippin.py -r test.txt
Initial package set:
- six ['>1']
- taskflow ['>1']
Traceback (most recent call last):
File "pippin.py", line 168, in<module>
main()
File "pippin.py", line 162, in main
matches = probe(initial, {})
File "pippin.py", line 139, in probe
result = probe(requirements, gathered)
File "pippin.py", line 129, in probe
m = find_match(pkg_name, req)
File "pippin.py", line 112, in find_match
return match_available(req.req, find_versions(pkg_name))
File "pippin.py", line 108, in match_available
" matches '%s' (tried %s)" % (req, looked_in))
__main__.NotFoundException: No requirement found that matches
'taskflow>1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
'0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])
I suspect all that is needed to add is the code that is marked with
FIXME/TODO there and this kind of recursive back-tracking might just do
the trick...
-Josh
Joe Gordon wrote:
On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon<joe.gord...@gmail.com
<mailto:joe.gord...@gmail.com>> wrote:
We can side step the dependency graphing and ordering issue by
looking at the list of curently installed packages via pip freeze
and not installing dependencies (pip install --no-deps)
After looking into this further here are the known issues:
* Partial capping won't work [0], so we need to pin all
dependencies, we can generate this list per file via "pip install
-r" and "pip freeze", but this doesn't address the issue of apt-get
vs pip install. For example in the stable gate we use suds 0.4.1 but
only suds 0.4.0 is available via pip.
* Not all packages are installed in are standard dsvm-tempest env,
so using pip-freeze from that isn't enough
* We need to run this per requirements file and move to using pip
install --no-deps everywhere. As the global-requirements sync
wouldn't work the first time since files don't list all transient
dependencies yet.
* We can still break if a package version is removed from pypi
* in pip-freeze we sometimes install versions lower then our minimum
version (python-libvirt!)
Exploring a few ideas here: https://review.openstack.org/#/c/147451/4
[0]
http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html
On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley<fu...@yuggoth.org
<mailto:fu...@yuggoth.org>> wrote:
On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
[...]
The other thing that happened was partial capping doesn't work,
because something else moves forward and breaks you from below. So
the patch will need to hit everything at once.
Right, and we _have_ to start using stable branches on all
clients/libraries to backport fixes as part of that. This means that
the stable branch management workflow is about to become pervasive
across some teams who were previously (blissfully?) ignorant of it.
Unresolved entirely is the tertiary dependencies (not direct
dependencies of any OpenStack project). That will need another
mechanism to seed them before any installation happens.
[...]
I won't go so far as to call it intractable, but I took a stab at it
about a year ago and building the dependency graph properly to be
able to do a depth-first ordering is nontrivial (enough that after
about a week hacking on possible solutions I gave up and switched to
more productive tasks). The primary complications I ran into were
identifying setup_requires in transitive dependencies and dealing
with platform/version-specific dependencies. That said, there's a
very good chance that more recent improvements in setuptools, pip
and virtualenv could make this task easier.
That's the things I can think off from the top of my head.
The implementation, from a devstack-gate perspective, is also going
to require a decision on whether we stick with stable/relname for
branches of libraries too or switch to some extended branch mapping
mechanism to be able to track stable/relnum branches for those. And
we're going to need more jobs to ensure that clients (specifically)
retain backward-compatibility from an appdev and end user
perspective since they'll no longer get any testing as server
dependencies on stable branches (due to being capped there).
--
Jeremy Stanley
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
