On Jul 7, 2014 4:48 PM, "Sean Dague" <[email protected]> wrote: > > This thread was unfortunately hidden under a project specific tag (I > have thus stripped all the tags). > > The crux of the argument here is the following: > > Is a stackforge project project able to propose additions to > global-requirements.txt that aren't used by any projects in OpenStack. > > I believe the answer is firmly *no*.
++ > > global-requirements.txt provides a way for us to have a single point of > vetting for requirements for OpenStack. It lets us assess licensing, > maturity, current state of packaging, python3 support, all in one place. > And it lets us enforce that integration of OpenStack projects all run > under a well understood set of requirements. > > The requirements sync that happens after requirements land is basically > just a nicety for getting openstack projects to the tested state by > eventual consistency. > > If a stackforge project wants to be limited by global-requirements, > that's cool. We have a mechanism for that. However, they are accepting > that they will be limited by it. That means they live with how the > OpenStack project establishes that list. It specifically means they > *don't* get to propose any new requirements. > > Basically in this case Solum wants to have it's cake and eat it to. Both > be enforced on requirements, and not be enforced. Or some 3rd thing that > means the same as that. > > The near term fix is to remove solum from projects.txt. The email included below mentions an additional motivation for using global-requirements is to avoid using pypi.python.org and instead use pypi.openstack.org for speed and reliability. Perhaps there is a way we can support use case for stackforge projects not in projects.txt? I thought I saw something the other day about adding a full pypi mirror to OpenStack infra. > > On 06/26/2014 02:00 AM, Adrian Otto wrote: > > Ok, > > > > I submitted and abandoned a couple of reviews[1][2] for a solution aimed > > to meet my goals without adding a new per-project requirements file. The > > flaw with this approach is that pip may install other requirements when > > installing the one(s) loaded from the fallback mirror, and those may > > conflict with the ones loaded from the primary mirror. > > > > After discussing this further in #openstack-infra this evening, we > > should give serious consideration to adding python-mistralclient to > > global requirements. I have posted a review[3] for that to get input > > from the requirements review team. > > > > Thanks, > > > > Adrian > > > > [1] https://review.openstack.org/102716 > > [2] https://review.openstack.org/102719 > > [3] https://review.openstack.org/102738 > > <https://review.openstack.org/1027387> > > > > On Jun 25, 2014, at 9:51 PM, Matthew Oliver <[email protected] > > <mailto:[email protected]>> wrote: > > > >> > >> On Jun 26, 2014 12:12 PM, "Angus Salkeld" <[email protected] > >> <mailto:[email protected]>> wrote: > >> > > > On 25/06/14 15:13, Clark Boylan wrote: > >> On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto > >>> <[email protected] <mailto:[email protected]>> wrote: > >>> Hello, > >>> > >>> Solum has run into a constraint with the current scheme for > >>> requirements management within the OpenStack CI system. We have a > >>> proposal for dealing with this constraint that involves making a > >>> contribution to openstack-infra. This message explains the constraint, > >>> and our proposal for addressing it. > >>> > >>> == Background == > >>> > >>> OpenStack uses a list of global requirements in the requirements > >>> repo[1], and each project has it’s own requirements.txt and > >>> test-requirements.txt files. The requirements are satisfied by gate > >>> jobs using pip configured to use the pypi.openstack.org > >>> <http://pypi.openstack.org/> mirror, which is periodically updated > >>> with new content from pypi.python.org <http://pypi.python.org/>. One > >>> motivation for doing this is that pypi.python.org > >>> <http://pypi.python.org/> may not be as fast or as reliable as a local > >>> mirror. The gate/check jobs for the projects use the OpenStack > >>> internal pypi mirror to ensure stability. > >>> > >>> The OpenStack CI system will sync up the requirements across all > >>> the official projects and will create reviews in the participating > >>> projects for any mis-matches. Solum is one of these projects, and > >>> enjoys this feature. > >>> > >>> Another motivation is so that users of OpenStack will have one > >>> single set of python package requirements/dependencies to install and > >>> run the individual OpenStack components. > >>> > >>> == Problem == > >>> > >>> Stackforge projects listed in openstack/requirements/projects.txt > >>> that decide to depend on each other (for example, Solum wanting to > >>> list mistralclient as a requirement) are unable to, because they are > >>> not yet integrated, and are not listed in > >>> openstack/requirements/global-requirements.txt yet. This means that in > >>> order to depend on each other, a project must withdraw from > >>> projects.txt and begin using pip with pypi.poython.org > >>> <http://pypi.poython.org/> to satisfy all of their requirements.I > >>> strongly dislike this option. > >>> > >>> Mistral is still evolving rapidly, and we don’t think it makes > >>> sense for them to pursue integration wight now. The upstream > >>> distributions who include packages to support OpenStack will also > >>> prefer not to deal with a requirement that will be cutting a new > >>> version every week or two in order to satisfy evolving needs as Solum > >>> and other consumers of Mistral help refine how it works. > >>> > >>> == Proposal == > >>> > >>> We want the best of both worlds. We want the freedom to innovate > >>> and use new software for a limited selection of stackforge projects, > >>> and still use the OpenStack pypi server to satisfy my regular > >>> requirements. We want the speed and reliability of using our local > >>> mirror, and users of Solum to use a matching set of requirements for > >>> all the things that we use, and integrated projects use. We want to > >>> continue getting the reviews that bring us up to date with new > >>> requirements versions. > >>> > >>> We propose that we submit an enhancement to the gate/check job > >>> setup that will: > >>> > >>> 1) Begin (as it does today) by satisfying global-requirements.txt > >>> and my local project’s requirements.txt and test-requirements.txt > >>> using the local OpenStack pypi mirror. > >>> 2) After all requirements are satisfied, check the name of my > >>> project. If it begins with ‘stackforge/‘ then look for a > >>> stackforge-requirements.txt file. If one exists, reconfigure pip to > >>> switch to use pypi.python.org <http://pypi.python.org/>, and satisfy > >>> the requirements listed in the file. We will list mistralclient there, > >>> and get the latest tagged/released version of that. > >>> > >> I am reasonably sure that if you remove yourself from the > >> openstack/requirements project list this is basically how it will > >> work. Pip is configured to use the OpenStack mirror and fall back on > >> pypi.python.org <http://pypi.python.org/> for packages not > >>> available on the OpenStack mirror > >> [2]. So I don't think there is any work to do here with additional > >> requirements files. It should just work. Adding a new requirements > >> file will just make things more confusing for packagers and consumers > >> of your software. > > > > Adrian I know this is not the optimal solution, but I think this is > > the most pragmatic solution (esp. given we need to progress and not > >>> be held > > up by this), most stackforge projects are in the same boat as us. > > As far as pypi breakages (most are easily fixable by restricting the > > package versions if we get an issue with a new release > > of *random-api-breaking-package*). > > > >>> > >>> I've looked through the infra choose mirror code, and Clark is > >>> correct. If the project isn't in the projects.txt file they will only > >>> access to pypi.openstack.org <http://pypi.openstack.org/> however if > >>> removed then it will first check pypi.openstack.org > >>> <http://pypi.openstack.org/> and then fall back to to pypi.python.org > >>> <http://pypi.python.org/>. I think the only real solution is what > >>> Angus mentioned, remove yourself from projects.txt at least until all > >>> your dependencies can be provided by pypi.openstack.org > >>> <http://pypi.openstack.org/> or another solution is put into place. In > >>> the mean time you can at least progress and continue development. > >>> > >>> If you code requires a direct dependency (rather then an optional > >>> dependency) of some non integrated project, then your stuck until they > >>> are. > >>> > >>> > > > >>> > >>> == Call To Action == > >>> > >>> What do you think of this approach to satisfy a balance of > >>> interests? Everything remains the same for OpenStack projects, and > >>> Stackforge projects get a new feature that allows them to require > >>> software that has not yet been integrated. Are there even better > >>> options that we should consider? > >>> > >>> Thanks, > >>> > >>> Adrian Otto > >>> > >>> > >>> References: > >>> [1] https://review.openstack.org/openstack/requirements > > > >> For what it is worth the Infra team has also been looking at > >> potentially using something like bandersnatch to mirror all of pypi > >> which is now a possibility because OpenStack doesn't depend on > >> packages that are hosted external to pypi. We would then do > >> requirements enforcement via checks rather than explicit use of a > >> restricted mirror. There are some things to sort out like platform > >> dependent wheels (I am not sure that any OpenStack project directly > >> consumes these but I have found them to be quite handy) and the > >> potential need for more enforcement to keep this working, but I think > >> this is a possibility. > > > > This would be neat. > > > > -Angus > > > > > >> Clark > > > >> [2] > >>> https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54 > > > >> _______________________________________________ > >> OpenStack-dev mailing list > >> [email protected] > >>> <mailto:[email protected]> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > >> > > >> > _______________________________________________ > >> > OpenStack-dev mailing list > >> > [email protected] > >> <mailto:[email protected]> > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/ > >> > >> _______________________________________________ > >> OpenStack-dev mailing list > >> [email protected] > >> <mailto:[email protected]> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > [email protected] > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > -- > Sean Dague > http://dague.net > > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
_______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
