On Tue, Jul 8, 2014 at 8:54 PM, James Polley <j...@jamezpolley.com> wrote:
> It may not have been clear from the below email, but clarkb clarifies on > https://bugs.launchpad.net/openstack-ci/+bug/1294381 that the infra team > is no longer maintaining pypi-mirror > > This has been a very useful tool for tripleo. It's much simpler for new > developers to set up and use than a full bandersnatch mirror (and requires > less disk space), and it can create a local cache of wheels which saves > build time. > > But it's now unsupported. > > To me it seems like we have two options: > > A) Deprecate usage of pypi-mirror; update docs to instruct new devs in > setting up a local bandersnatch mirror instead > or > B) Take on care-and-feeding of the tool. > or, I guess, > C) Continue to recommend people use an unsupported unmaintained > known-buggy tool (it works reasonably well for us today, but it's going to > work less and less well as time goes by) > > Are there other options I haven't thought of? > I don't know if this fits your requirements but I use http://doc.devpi.net/latest/quickstart-pypimirror.html for my development needs. > > Do you have thoughts on which option is preferred? > > > ---------- Forwarded message ---------- > From: Clark Boylan <clark.boy...@gmail.com> > Date: Tue, Jul 8, 2014 at 8:50 AM > Subject: Re: [openstack-dev] Policy around Requirements Adds (was: New > class of requirements for Stackforge projects) > To: "OpenStack Development Mailing List (not for usage questions)" < > openstack-dev@lists.openstack.org> > > > On Mon, Jul 7, 2014 at 3:45 PM, Joe Gordon <joe.gord...@gmail.com> wrote: > > > > On Jul 7, 2014 4:48 PM, "Sean Dague" <s...@dague.net> wrote: > >> > >> This thread was unfortunately hidden under a project specific tag (I > >> have thus stripped all the tags). > >> > >> The crux of the argument here is the following: > >> > >> Is a stackforge project project able to propose additions to > >> global-requirements.txt that aren't used by any projects in OpenStack. > >> > >> I believe the answer is firmly *no*. > > > > ++ > > > >> > >> global-requirements.txt provides a way for us to have a single point of > >> vetting for requirements for OpenStack. It lets us assess licensing, > >> maturity, current state of packaging, python3 support, all in one place. > >> And it lets us enforce that integration of OpenStack projects all run > >> under a well understood set of requirements. > >> > >> The requirements sync that happens after requirements land is basically > >> just a nicety for getting openstack projects to the tested state by > >> eventual consistency. > >> > >> If a stackforge project wants to be limited by global-requirements, > >> that's cool. We have a mechanism for that. However, they are accepting > >> that they will be limited by it. That means they live with how the > >> OpenStack project establishes that list. It specifically means they > >> *don't* get to propose any new requirements. > >> > >> Basically in this case Solum wants to have it's cake and eat it to. Both > >> be enforced on requirements, and not be enforced. Or some 3rd thing that > >> means the same as that. > >> > >> The near term fix is to remove solum from projects.txt. > > > > The email included below mentions an additional motivation for using > > global-requirements is to avoid using pypi.python.org and instead use > > pypi.openstack.org for speed and reliability. Perhaps there is a way we > can > > support use case for stackforge projects not in projects.txt? I thought I > > saw something the other day about adding a full pypi mirror to OpenStack > > infra. > > > This is done. All tests are now run against a bandersnatch built full > mirror of pypi. Enforcement of the global requirements is performed > via the enforcement jobs. > >> > >> On 06/26/2014 02:00 AM, Adrian Otto wrote: > >> > Ok, > >> > > >> > I submitted and abandoned a couple of reviews[1][2] for a solution > aimed > >> > to meet my goals without adding a new per-project requirements file. > The > >> > flaw with this approach is that pip may install other requirements > when > >> > installing the one(s) loaded from the fallback mirror, and those may > >> > conflict with the ones loaded from the primary mirror. > >> > > >> > After discussing this further in #openstack-infra this evening, we > >> > should give serious consideration to adding python-mistralclient to > >> > global requirements. I have posted a review[3] for that to get input > >> > from the requirements review team. > >> > > >> > Thanks, > >> > > >> > Adrian > >> > > >> > [1] https://review.openstack.org/102716 > >> > [2] https://review.openstack.org/102719 > >> > [3] https://review.openstack.org/102738 > >> > <https://review.openstack.org/1027387> > >> > > >> > On Jun 25, 2014, at 9:51 PM, Matthew Oliver <m...@oliver.net.au > >> > <mailto:m...@oliver.net.au>> wrote: > >> > > >> >> > >> >> On Jun 26, 2014 12:12 PM, "Angus Salkeld" < > angus.salk...@rackspace.com > >> >> <mailto:angus.salk...@rackspace.com>> wrote: > >> >> > > >> > On 25/06/14 15:13, Clark Boylan wrote: > >> >> On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto > >> >>> <adrian.o...@rackspace.com <mailto:adrian.o...@rackspace.com>> > wrote: > >> >>> Hello, > >> >>> > >> >>> Solum has run into a constraint with the current scheme for > >> >>> requirements management within the OpenStack CI system. We have a > >> >>> proposal for dealing with this constraint that involves making a > >> >>> contribution to openstack-infra. This message explains the > constraint, > >> >>> and our proposal for addressing it. > >> >>> > >> >>> == Background == > >> >>> > >> >>> OpenStack uses a list of global requirements in the requirements > >> >>> repo[1], and each project has it’s own requirements.txt and > >> >>> test-requirements.txt files. The requirements are satisfied by gate > >> >>> jobs using pip configured to use the pypi.openstack.org > >> >>> <http://pypi.openstack.org/> mirror, which is periodically updated > >> >>> with new content from pypi.python.org <http://pypi.python.org/>. > One > >> >>> motivation for doing this is that pypi.python.org > >> >>> <http://pypi.python.org/> may not be as fast or as reliable as a > local > >> >>> mirror. The gate/check jobs for the projects use the OpenStack > >> >>> internal pypi mirror to ensure stability. > >> >>> > >> >>> The OpenStack CI system will sync up the requirements across all > >> >>> the official projects and will create reviews in the participating > >> >>> projects for any mis-matches. Solum is one of these projects, and > >> >>> enjoys this feature. > >> >>> > >> >>> Another motivation is so that users of OpenStack will have one > >> >>> single set of python package requirements/dependencies to install > and > >> >>> run the individual OpenStack components. > >> >>> > >> >>> == Problem == > >> >>> > >> >>> Stackforge projects listed in openstack/requirements/projects.txt > >> >>> that decide to depend on each other (for example, Solum wanting to > >> >>> list mistralclient as a requirement) are unable to, because they are > >> >>> not yet integrated, and are not listed in > >> >>> openstack/requirements/global-requirements.txt yet. This means that > in > >> >>> order to depend on each other, a project must withdraw from > >> >>> projects.txt and begin using pip with pypi.poython.org > >> >>> <http://pypi.poython.org/> to satisfy all of their requirements.I > >> >>> strongly dislike this option. > >> >>> > >> >>> Mistral is still evolving rapidly, and we don’t think it makes > >> >>> sense for them to pursue integration wight now. The upstream > >> >>> distributions who include packages to support OpenStack will also > >> >>> prefer not to deal with a requirement that will be cutting a new > >> >>> version every week or two in order to satisfy evolving needs as > Solum > >> >>> and other consumers of Mistral help refine how it works. > >> >>> > >> >>> == Proposal == > >> >>> > >> >>> We want the best of both worlds. We want the freedom to innovate > >> >>> and use new software for a limited selection of stackforge projects, > >> >>> and still use the OpenStack pypi server to satisfy my regular > >> >>> requirements. We want the speed and reliability of using our local > >> >>> mirror, and users of Solum to use a matching set of requirements for > >> >>> all the things that we use, and integrated projects use. We want to > >> >>> continue getting the reviews that bring us up to date with new > >> >>> requirements versions. > >> >>> > >> >>> We propose that we submit an enhancement to the gate/check job > >> >>> setup that will: > >> >>> > >> >>> 1) Begin (as it does today) by satisfying global-requirements.txt > >> >>> and my local project’s requirements.txt and test-requirements.txt > >> >>> using the local OpenStack pypi mirror. > >> >>> 2) After all requirements are satisfied, check the name of my > >> >>> project. If it begins with ‘stackforge/‘ then look for a > >> >>> stackforge-requirements.txt file. If one exists, reconfigure pip to > >> >>> switch to use pypi.python.org <http://pypi.python.org/>, and > satisfy > >> >>> the requirements listed in the file. We will list mistralclient > there, > >> >>> and get the latest tagged/released version of that. > >> >>> > >> >> I am reasonably sure that if you remove yourself from the > >> >> openstack/requirements project list this is basically how it will > >> >> work. Pip is configured to use the OpenStack mirror and fall back on > >> >> pypi.python.org <http://pypi.python.org/> for packages not > >> >>> available on the OpenStack mirror > >> >> [2]. So I don't think there is any work to do here with additional > >> >> requirements files. It should just work. Adding a new requirements > >> >> file will just make things more confusing for packagers and consumers > >> >> of your software. > >> > > >> > Adrian I know this is not the optimal solution, but I think this is > >> > the most pragmatic solution (esp. given we need to progress and not > >> >>> be held > >> > up by this), most stackforge projects are in the same boat as us. > >> > As far as pypi breakages (most are easily fixable by restricting the > >> > package versions if we get an issue with a new release > >> > of *random-api-breaking-package*). > >> > > >> >>> > >> >>> I've looked through the infra choose mirror code, and Clark is > >> >>> correct. If the project isn't in the projects.txt file they will > only > >> >>> access to pypi.openstack.org <http://pypi.openstack.org/> however > if > >> >>> removed then it will first check pypi.openstack.org > >> >>> <http://pypi.openstack.org/> and then fall back to to > pypi.python.org > >> >>> <http://pypi.python.org/>. I think the only real solution is what > >> >>> Angus mentioned, remove yourself from projects.txt at least until > all > >> >>> your dependencies can be provided by pypi.openstack.org > >> >>> <http://pypi.openstack.org/> or another solution is put into > place. In > >> >>> the mean time you can at least progress and continue development. > >> >>> > >> >>> If you code requires a direct dependency (rather then an optional > >> >>> dependency) of some non integrated project, then your stuck until > they > >> >>> are. > >> >>> > >> >>> > >> > > >> >>> > >> >>> == Call To Action == > >> >>> > >> >>> What do you think of this approach to satisfy a balance of > >> >>> interests? Everything remains the same for OpenStack projects, and > >> >>> Stackforge projects get a new feature that allows them to require > >> >>> software that has not yet been integrated. Are there even better > >> >>> options that we should consider? > >> >>> > >> >>> Thanks, > >> >>> > >> >>> Adrian Otto > >> >>> > >> >>> > >> >>> References: > >> >>> [1] https://review.openstack.org/openstack/requirements > >> > > >> >> For what it is worth the Infra team has also been looking at > >> >> potentially using something like bandersnatch to mirror all of pypi > >> >> which is now a possibility because OpenStack doesn't depend on > >> >> packages that are hosted external to pypi. We would then do > >> >> requirements enforcement via checks rather than explicit use of a > >> >> restricted mirror. There are some things to sort out like platform > >> >> dependent wheels (I am not sure that any OpenStack project directly > >> >> consumes these but I have found them to be quite handy) and the > >> >> potential need for more enforcement to keep this working, but I think > >> >> this is a possibility. > >> > > >> > This would be neat. > >> > > >> > -Angus > >> > > >> > > >> >> Clark > >> > > >> >> [2] > >> >>> > >> >>> > https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54 > >> > > >> >> _______________________________________________ > >> >> OpenStack-dev mailing list > >> >> OpenStack-dev@lists.openstack.org > >> >>> <mailto:OpenStack-dev@lists.openstack.org> > >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > >> > > >> >> > > >> >> > _______________________________________________ > >> >> > OpenStack-dev mailing list > >> >> > OpenStack-dev@lists.openstack.org > >> >> <mailto:OpenStack-dev@lists.openstack.org> > >> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/ > >> >> > >> >> _______________________________________________ > >> >> OpenStack-dev mailing list > >> >> OpenStack-dev@lists.openstack.org > >> >> <mailto:OpenStack-dev@lists.openstack.org> > >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > >> > > >> > > >> > _______________________________________________ > >> > OpenStack-dev mailing list > >> > OpenStack-dev@lists.openstack.org > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > >> > >> -- > >> Sean Dague > >> http://dague.net > >> > >> > >> _______________________________________________ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev