Re: Trying to outline the steps taken to go from "I want this package" to it being installed
Brett, did you end up making progress on this? If not, would you be open to someone else picking it up? Thanks! -- Sumana Harihareswara Changeset Consulting https://changeset.nyc On 3/5/18 1:01 PM, Brett Cannon wrote: > Thanks for the extra details, Nick! I have some documentation to read on > some projects now that I have a complete list, but once that's done I'll > come back here with my idea. ;) > > On Fri, 2 Mar 2018 at 21:50 Nick Coghlan wrote: > >> On 3 March 2018 at 06:55, Brett Cannon wrote: >> >>> I have a project idea, but before I start it I need to make sure that I >>> have the high-order steps necessary to go from `pip install pip=9.0.1` to >>> it actually ending up on disk. Now I'm only considered with >>> modern/bleeding-edge, spec-based stuff, so PEP 517/518 and no setup.py, etc. >>> >>> Anyway, if people can point out any steps the below outline is missing I >>> would appreciate it. Thanks! >>> >>> >>>1. Specify package requirement >>> 1. Translate name to PyPI-compatible name >>> 2. Tease out requirement details (e.g. version, markers, etc.) >>>2. Check if package is already installed >>> >>> >> Depending on the installer design, a local download/build cache may be >> checked before checking PyPI (and since you include a caching step later, >> you'll presumably want to cover the caching step as well). >> >> >>> >>>1. Check PyPI for package >>>2. Choose appropriate file >>> 1. Get list of files >>> 2. Calculate best-fitting wheel >>> 3. Fallback to .tar.gz sdist >>>3. Download file >>>4. If sdist: >>> 1. Extract >>> 2. Read pyproject.toml >>> 3. Create venv >>> 4. Install build dependencies >>> >>> >> After installing the static build dependencies, you also need to query for >> any dynamic build dependencies and install them if they're requested: >> https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel >> >> This build dependency installation step can get arbitrarily complicated if >> you allow build dependencies to be installed from source, so the initial >> implementation in pip requires that build dependencies already be available >> as wheel files (either on the index server or in the local artifact cache). >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >> >
Re: PyPI JSON API redirect loop for all unpublished packages
Donald and Dustin: have we been running into these kinds of Travis problems in the past few months or does it seem to have settled down? -- Sumana Harihareswara Changeset Consulting https://changeset.nyc On 5/19/18 4:35 PM, Brett Cannon wrote: > On Fri, May 18, 2018, 06:08 Dustin Ingram, wrote: > >> I did reach out to the one contact we had there from when GCP/Fastly >> were having issues that affected Travis/PyPI (Emma) on Monday, but got >> no response. >> > > If Travis doesn't work out then let Steve Dower and me know and we can see > if we can get you extra credits on VSTS (teammate of mine was already > working with Jason at the PyCon sprints to get setuptools up on > PyPA.visualstudio.com). > > -Brett > > >> D. >> >> On Fri, May 18, 2018 at 8:54 AM, Sumana Harihareswara >> wrote: >>> In my opinion, this kind of bottleneck is likely to happen more >> frequently as we increase PyPA development activity, so it'd be worth >> asking Travis to bump up our account's oomph. (I am very tired and on a >> train so please forgive handwavy wording.) Do we have a contact there? >>> >>> -- >>> Sumana Harihareswara >>> Changeset Consulting >>> https://changeset.nyc >>> >>> On 05/16/2018 04:36 PM, Dustin Ingram wrote: Thanks for the report. I reverted the commit that caused this in https://github.com/pypa/warehouse/pull/4015, however it might take a bit for this to get deployed to PyPI because there's currently a pretty long backlog in Travis due to all the PyPA development happening during the sprints. D.
Re: Documentation on running Warehouse in your own production evironment?
Hi and thanks for writing! And thanks for being clear and comprehensive about what you are looking for. I'm sorry you didn't come across https://warehouse.readthedocs.io/application/#usage-assumptions-and-concepts -- maybe we need to flag that better. As it mentions: > Warehouse is specifically the codebase for the official Python Package Index, > and thus focuses on architecture and features for PyPI and Test PyPI. People > and groups who want to run their own package indexes usually use other tools, > like devpi https://pypi.org/project/devpi-server/ . You might also consider https://github.com/pypiserver/pypiserver or one of the other similar projects: https://github.com/pypiserver/pypiserver#similar-projects I hope this helps! (Sorry, I originally (yesterday) sent this off-list by mistake.) -- Sumana Harihareswara Changeset Consulting https://changeset.nyc On 12/20/18 9:23 AM, Christoph Bischko wrote: > Hi all, > > currently, the only official warehouse documentation at > https://warehouse.readthedocs.io seems to be quite sparse. Also it seems to > be aimed at developers, not end users. A lot about features and the hows of > setting up a warehouse instance is left in the dark. > > For instance, there is the environment file > (https://github.com/pypa/warehouse/blob/master/dev/environment) that > contains some very necessary settings. I did not find out about it in the > documentation but by a painful reading of the Makefile. > Warehouse ships with "example data". I.e. on installation there are >40k > users and >30k dummy packages in the database by default - no instructions > on how to remove them. Again, I read the Makefile and altered the > example.sql database as a result. > HTTPS seems to be disabled by default, with no documentation on how to set > it up properly and securely. > Account verification Mails, do not work out of the box - again no docs on > setting that up. > Pip installation of packages on a client via the index on my local > Warehouse failed, because the links pointed to local host instead of the > actual fileserver. Again, the responsible setting FILES_BACKEND in the > environment is nowhere to be read about. > There is no information on whether it is possible and how, to set up > caching of the index at pypi.org with a local warehouse. > > As you can see, I was able to resolve some of these issues with some > digging, reading of the code, trial and error and a bit pain. But my > questions are: > > - Am I missing something here? Is warehouse not meant to be used in > setting up your own local package index, i.e. for a company or educational > facility? > - Is there additional documentation I'm not aware of, something aimed at > system administrators and end-users that want to setup their own pypi, > because searching the web gives precious little and nothing usable? > - Are there (maybe 3rd-party) example configurations and guides for > warehouse? > > The goal for a local instance of warehouse would be: > - Isolation of the local network from the internet (i.e. caching of pypi) > - Speedup of package installation via local network > - Having private packages locally that are not uploaded to pypi > > I hope you can help me, and maybe documenting warehouse for end users will > get a higher priority as a result. > Thanks, > Christoph > > >
