I'll follow-up on the spec, but one thing Donald has been pointing out for a while is that we don't use requirements.txt the way that pip anticipates: the expected use is that a specific install (e.g. the gate) will have a very specific list of requirements, caps etc, but that the install_requires will be as minimal as possible to ensure the project builds and self-tests ok.
I see the issues here as being related to that. -Rob On 26 February 2015 at 10:04, Joe Gordon <[email protected]> wrote: > > > On Tue, Feb 24, 2015 at 7:00 AM, Doug Hellmann <[email protected]> > wrote: >> >> >> >> On Mon, Feb 23, 2015, at 06:31 PM, Joe Gordon wrote: >> > On Mon, Feb 23, 2015 at 11:04 AM, Doug Hellmann <[email protected]> >> > wrote: >> > >> > > >> > > >> > > On Mon, Feb 23, 2015, at 12:26 PM, Joe Gordon wrote: >> > > > On Mon, Feb 23, 2015 at 8:49 AM, Ihar Hrachyshka >> > > > <[email protected]> >> > > > wrote: >> > > > >> > > > > -----BEGIN PGP SIGNED MESSAGE----- >> > > > > Hash: SHA1 >> > > > > >> > > > > On 02/20/2015 07:16 PM, Joshua Harlow wrote: >> > > > > > Sean Dague wrote: >> > > > > >> On 02/20/2015 12:26 AM, Adam Gandelman wrote: >> > > > > >>> Its more than just the naming. In the original proposal, >> > > > > >>> requirements.txt is the compiled list of all pinned deps >> > > > > >>> (direct and transitive), while >> > > > > >>> requirements.in<http://requirements.in> reflects what people >> > > > > >>> will actually use. Whatever is in requirements.txt affects >> > > > > >>> the >> > > > > >>> egg's requires.txt. Instead, we can keep requirements.txt >> > > > > >>> unchanged and have it still be the canonical list of >> > > > > >>> dependencies, while >> > > > > >>> reqiurements.out/requirements.gate/requirements.whatever is an >> > > > > >>> upstream utility we produce and use to keep things sane on our >> > > > > >>> slaves. >> > > > > >>> >> > > > > >>> Maybe all we need is: >> > > > > >>> >> > > > > >>> * update the existing post-merge job on the requirements repo >> > > > > >>> to produce a requirements.txt (as it does now) as well the >> > > > > >>> compiled version. >> > > > > >>> >> > > > > >>> * modify devstack in some way with a toggle to have it process >> > > > > >>> dependencies from the compiled version when necessary >> > > > > >>> >> > > > > >>> I'm not sure how the second bit jives with the existing >> > > > > >>> devstack installation code, specifically with the libraries >> > > > > >>> from git-or-master but we can probably add something to warm >> > > > > >>> the system with dependencies from the compiled version prior >> > > > > >>> to >> > > > > >>> calling pip/setup.py/etc <http://setup.py/etc> >> > > > > >> >> > > > > >> It sounds like you are suggesting we take the tool we use to >> > > > > >> ensure that all of OpenStack is installable together in a >> > > > > >> unified >> > > > > >> way, and change it's installation so that it doesn't do that >> > > > > >> any >> > > > > >> more. >> > > > > >> >> > > > > >> Which I'm fine with. >> > > > > >> >> > > > > >> But if we are doing that we should just whole hog give up on >> > > > > >> the >> > > > > >> idea that OpenStack can be run all together in a single >> > > > > >> environment, and just double down on the devstack venv work >> > > > > >> instead. >> > > > > > >> > > > > > It'd be interesting to see what a distribution (canonical, >> > > > > > redhat...) would think about this movement. I know yahoo! has >> > > > > > been >> > > > > > looking into it for similar reasons (but we are more flexibly >> > > > > > then >> > > > > > I think a packager such as canonical/redhat/debian/... >> > > > > > would/culd >> > > > > > be). With a move to venv's that seems like it would just offload >> > > > > > the work to find the set of dependencies that work together (in >> > > > > > a >> > > > > > single-install) to packagers instead. >> > > > > > >> > > > > > Is that ok/desired at this point? >> > > > > > >> > > > > >> > > > > Honestly, I failed to track all the different proposals. Just >> > > > > saying >> > > > > from packager perspective: we absolutely rely on requirements.txt >> > > > > not >> > > > > being a list of hardcoded values from pip freeze, but present us a >> > > > > reasonable freedom in choosing versions we want to run in packaged >> > > > > products. >> > > > > >> > > > > >> > > > in short the current proposal for stable branches is: >> > > > >> > > > keep requirements.txt as is, except maybe put some upper bounds on >> > > > the >> > > > requirements. >> > > > >> > > > Add requirements.gate to specify the *exact* versions we are gating >> > > > against >> > > > (this would be a full list including all transitive dependencies). >> > > >> > > The gate syncs requirements into projects before installing them. >> > > Would >> > > we change the sync script for the gate to work from the >> > > requirements.gate file, or keep it pulling from requirements.txt? >> > > >> > >> > We would only add requirements.gate for stable branches (because we >> > don't >> > want to cap/pin things on master). So I think the answer is sync script >> > should work for both. I am not sure on the exact mechanics of how this >> > would work. Whoever ends up driving this bit of work (I think Adam G), >> > will >> > have to sort out the details. >> >> OK. I think it's probably worth a spec, then, so we can think it through >> before starting work. Maybe in the cross-project specs repo, to avoid >> having to create one just for requirements? Or we could modify the >> README or something, but the specs repo seems more visible. > > > Start of the cross project spec https://review.openstack.org/159249 > >> >> >> Doug >> >> > >> > >> > > Doug >> > > >> > > > >> > > > >> > > > > That's why I asked before we should have caps and not pins. >> > > > > >> > > > > /Ihar >> > > > > -----BEGIN PGP SIGNATURE----- >> > > > > Version: GnuPG v1 >> > > > > >> > > > > iQEcBAEBAgAGBQJU61oJAAoJEC5aWaUY1u57T7cIALySnlpLV0tjrsTH2gZxskH+ >> > > > > zY+L6E/DukFNZsWxB2XSaOuVdVaP3Oj4eYCZ2iL8OoxLrBotiOYyRFH29f9vjNSX >> > > > > h++dErBr0SwIeUtcnEjbk9re6fNP6y5Hqhk1Ac+NSxwL75KlS3bgKnJAhLA08MVB >> > > > > 5xkGRR7xl2cuYf9eylPlQaAy9rXPCyyRdxZs6mNjZ2vlY6hZx/w/Y7V28R/V4gO4 >> > > > > qsvMg6Kv+3urDTRuJdEsV6HbN/cXr2+o543Unzq7gcPpDYXRFTLkpCRV2k8mnmA1 >> > > > > pO9W10F1FCQZiBnLk0c6OypFz9rQmKxpwlNUN5MTMF15Et6DOxGBxMcfr7TpRaQ= >> > > > > =WHOH >> > > > > -----END PGP SIGNATURE----- >> > > > > >> > > > > >> > > >> > > __________________________________________________________________________ >> > > > > OpenStack Development Mailing List (not for usage questions) >> > > > > Unsubscribe: >> > > [email protected]?subject:unsubscribe >> > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > > >> > > > >> > > >> > > __________________________________________________________________________ >> > > > OpenStack Development Mailing List (not for usage questions) >> > > > Unsubscribe: >> > > > [email protected]?subject:unsubscribe >> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > >> > > >> > > __________________________________________________________________________ >> > > OpenStack Development Mailing List (not for usage questions) >> > > Unsubscribe: >> > > [email protected]?subject:unsubscribe >> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > >> > >> > __________________________________________________________________________ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: >> > [email protected]?subject:unsubscribe >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: [email protected]?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Robert Collins <[email protected]> Distinguished Technologist HP Converged Cloud __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
