Another thing that I just started whipping together:

https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f

The idea for the above is to use pip to download dependencies, but figure out what versions will work using our own resolver (and our own querying of 'http://pypi.python.org/pypi/%s/json') that just does a very deep search of all requirements (and requirements of requirements...).

The idea for that is that the probe() function in that gist will 'freeze' a single requirement then dive down into further requirements and ensure compatibility while that 'diving' (aka, recursion into further requirements) is underway. If a incompatibility is found then the recursion will back-track and try a to freeze a different version of a desired package (and repeat...).

To me this kind of deep finding would be a potential way of making this work in a way that basically only uses pip for downloading (and does the deep matching/probing) on our own since once the algorithm above doesn't backtrack and finds a matching set of requirements that will all work together the program can exit (and this set can then be used as the master set for openstack; at that point we might have to tell people to not use pip, or to only use pip --download to fetch the compatible versions).

It's not completed but it could be complementary to what others are working on; feel free to hack away :)

So far the following works:

$ cat test.txt
six>1
taskflow>1

$ python pippin.py  -r test.txt
Initial package set:
- six ['>1']
- taskflow ['>1']
Traceback (most recent call last):
  File "pippin.py", line 168, in <module>
    main()
  File "pippin.py", line 162, in main
    matches = probe(initial, {})
  File "pippin.py", line 139, in probe
    result = probe(requirements, gathered)
  File "pippin.py", line 129, in probe
    m = find_match(pkg_name, req)
  File "pippin.py", line 112, in find_match
    return match_available(req.req, find_versions(pkg_name))
  File "pippin.py", line 108, in match_available
    " matches '%s' (tried %s)" % (req, looked_in))
__main__.NotFoundException: No requirement found that matches 'taskflow>1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21', '0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])

I suspect all that is needed to add is the code that is marked with FIXME/TODO there and this kind of recursive back-tracking might just do the trick...

-Josh

Joe Gordon wrote:


On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon <joe.gord...@gmail.com
<mailto:joe.gord...@gmail.com>> wrote:

    We can side step the dependency graphing and ordering issue by
    looking at the list of curently installed packages via pip freeze
    and not installing dependencies (pip install --no-deps)

    After looking into this further here are the known issues:

    * Partial capping won't work [0], so we need to pin all
    dependencies, we can generate this list per file via "pip install
    -r" and "pip freeze", but this doesn't address the issue of apt-get
    vs pip install. For example in the stable gate we use suds 0.4.1 but
    only suds 0.4.0 is available via pip.
    * Not all packages are installed in are standard dsvm-tempest env,
    so using pip-freeze from that isn't enough
    * We need to run this per requirements file and move to using pip
    install --no-deps everywhere. As the global-requirements sync
    wouldn't work the first time since files don't list all transient
    dependencies yet.
    * We can still break if a package version is removed from pypi
    * in pip-freeze we sometimes install versions lower then our minimum
    version (python-libvirt!)


Exploring a few ideas here: https://review.openstack.org/#/c/147451/4



    [0]
    http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html

    On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley <fu...@yuggoth.org
    <mailto:fu...@yuggoth.org>> wrote:

        On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
        [...]
        >  The other thing that happened was partial capping doesn't work,
        >  because something else moves forward and breaks you from below. So
        >  the patch will need to hit everything at once.

        Right, and we _have_ to start using stable branches on all
        clients/libraries to backport fixes as part of that. This means that
        the stable branch management workflow is about to become pervasive
        across some teams who were previously (blissfully?) ignorant of it.

        >  Unresolved entirely is the tertiary dependencies (not direct
        >  dependencies of any OpenStack project). That will need another
        >  mechanism to seed them before any installation happens.
        [...]

        I won't go so far as to call it intractable, but I took a stab at it
        about a year ago and building the dependency graph properly to be
        able to do a depth-first ordering is nontrivial (enough that after
        about a week hacking on possible solutions I gave up and switched to
        more productive tasks). The primary complications I ran into were
        identifying setup_requires in transitive dependencies and dealing
        with platform/version-specific dependencies. That said, there's a
        very good chance that more recent improvements in setuptools, pip
        and virtualenv could make this task easier.

        >  That's the things I can think off from the top of my head.

        The implementation, from a devstack-gate perspective, is also going
        to require a decision on whether we stick with stable/relname for
        branches of libraries too or switch to some extended branch mapping
        mechanism to be able to track stable/relnum branches for those. And
        we're going to need more jobs to ensure that clients (specifically)
        retain backward-compatibility from an appdev and end user
        perspective since they'll no longer get any testing as server
        dependencies on stable branches (due to being capped there).
        --
        Jeremy Stanley

        
__________________________________________________________________________
        OpenStack Development Mailing List (not for usage questions)
        Unsubscribe:
        openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
        <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
        http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to