[Distutils] Re: packaging guide and private packages index
On 31 July 2018 at 08:13, Paul Moore wrote: > On 30 July 2018 at 23:01, David Cournapeau wrote: >> Hi, >> >> I see that there is almost no mention of private packages index in the >> packaging guide, and no recommendation on what to use. >> >> Currently googling for private packages mostly return obsolete (and not very >> practical) recommendations based on dependency links. >> >> In 2018, what would be the recommended practices for teams that develop >> private packages that depend on each other ? > > That's a very good point, and I think it would be a really useful > addition to the packaging guide. I believe the most commonly > recommended approach is to use a local,devpi instance as a private > index, but in terms of development workflow around that basis, I don't > really know that there's much in the way of a consensus. (I should say > that I don't work in an environment like this myself, so my comments > are based purely on what I've heard, not on personal experience). Ah, I thought we already had something on this: https://packaging.python.org/guides/index-mirrors-and-caches/ That emphasises the devpi caching proxy use case though, and doesn't point out that devpi is actually a full-fledged private Python package repository manager that's still lightweight enough to run locally as a caching proxy: https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html So even with that page, there's likely value in adding a second guide called something like "Running a private index server" that covers the options from "Just use a simple static file server", through to a Python specific package management engine like devpi, and on to open source multi-format repository management systems like Pulp and the various commercial options (I don't think we should be overly averse to mentioning those - we're mentioning commercial deployment platforms in the packaging overview, and honestly, there are a lot of folks out there where one of the professional lessons they're still learning is "your time as a developer is expensive, so you typically don't want to build what you can already buy"). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/5KSCJ44O43JB4SKIVO5FIQYL2NOIM5TS/
[Distutils] Re: packaging guide and private packages index
> On Jul 31, 2018, at 12:38 PM, Alex Becker wrote: > > Packagecloud only supports the "simple" index API, which makes it slower for > tools to work with and presumably less future-proof. It's particularly > annoying to me, since I've been working on a dependency management system and > only added support for PyPI's new JSON API. The simple API is the standard API for dependency fetching, while the same information exists in the JSON API, the JSON API is not standardized so alternative tools (both client side and server side) generally can’t implement it without just trying to mimic whatever PyPI does, whereas simple has a documented standard and changes to it require going through change management via the PEP process.-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/QH2DNMZNLGRXYWYTFMV7WFHPWA6WVZUH/
[Distutils] Re: packaging guide and private packages index
My company was recently in this situation, and it was definitely hard to get information on non-self-hosted solutions for private packages. We settled on packagecloud.io, but there are some definite pain points that I think PyPA could productively document: - Many tools do not support indices other than PyPI well. It is cumbersome and poorly documented to specify another index, and that index is only ever used as a fallback, increasing runtime and creating name-squatting concerns. - It is unclear which "python packaging" services actually offer a PyPI-like index, as opposed to an orthogonal Python library distribution strategy. - Packagecloud only supports the "simple" index API, which makes it slower for tools to work with and presumably less future-proof. It's particularly annoying to me, since I've been working on a dependency management system and only added support for PyPI's new JSON API. Having a quick survey of the options (including commercial ones) available and how tools integrate with them would have been very useful to us. On Tue, Jul 31, 2018 at 7:44 AM David Cournapeau wrote: > > > On Tue, Jul 31, 2018 at 7:44 AM Nathaniel Smith wrote: > >> On Mon, Jul 30, 2018 at 3:13 PM, Paul Moore wrote: >> > On 30 July 2018 at 23:01, David Cournapeau wrote: >> >> Hi, >> >> >> >> I see that there is almost no mention of private packages index in the >> >> packaging guide, and no recommendation on what to use. >> >> >> >> Currently googling for private packages mostly return obsolete (and >> not very >> >> practical) recommendations based on dependency links. >> >> >> >> In 2018, what would be the recommended practices for teams that develop >> >> private packages that depend on each other ? >> > >> > That's a very good point, and I think it would be a really useful >> > addition to the packaging guide. I believe the most commonly >> > recommended approach is to use a local,devpi instance as a private >> > index, but in terms of development workflow around that basis, I don't >> > really know that there's much in the way of a consensus. (I should say >> > that I don't work in an environment like this myself, so my comments >> > are based purely on what I've heard, not on personal experience). >> >> A tricky aspect here is that this niche has been colonized by >> commercial projects like Artifactory, Gemfury, etc., and I'm not sure >> we want to get into recommending one commercial product versus another >> (or that we even have any effective way to compare them). >> > > Right, I would not expect us to do anything besides maybe mentioning > proprietary solutions. > > But for OSS solutions, there are multiple options, and it is not clear > which ones are appropriate for which usage. E.g. do you need multi index > support, do you need upload support, etc. ? > > Would it make sense to start such a document as a PR ? > > David > > >> -n >> >> -- >> Nathaniel J. Smith -- https://vorpus.org >> > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/7E3UF7J3TDVHG7DKBGJZF627D4JSDM7K/ > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/4L62CFN6J5L24RGJTXN46U3IJ7HUPZTJ/
[Distutils] Re: packaging guide and private packages index
I was able to point pip at an instance of nginx that was configured to serve up a directory listing. Any gzipped tarballs of python source created by sdist would get served such that python would use them. On Mon, Jul 30, 2018 at 3:16 PM Wayne Werner wrote: > When I was in that situation I just reverse engineered a very *very* > simple PyPi clone: > > https://github.com/waynew/flaskpypi > > There aren't actually any docs, but you just have to put your packages in > a specific directory and it will serve them up for you. You just have to > tell pip to use the url for that index. If it doesn't have the > package/version locally then it will check the proper pypi for it. > > HTH, > Wayne > === > I welcome VSRE emails. Learn more at http://vsre.info/ > === > > > On Mon, Jul 30, 2018 at 5:04 PM David Cournapeau > wrote: > >> Hi, >> >> I see that there is almost no mention of private packages index in the >> packaging guide, and no recommendation on what to use. >> >> Currently googling for private packages mostly return obsolete (and not >> very practical) recommendations based on dependency links. >> >> In 2018, what would be the recommended practices for teams that develop >> private packages that depend on each other ? >> >> thank you, >> >> David >> -- >> Distutils-SIG mailing list -- distutils-sig@python.org >> To unsubscribe send an email to distutils-sig-le...@python.org >> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ >> Message archived at >> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/QLDBOAJZOZACT3FGHFC664DLHOHMBLDT/ >> > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/7YMKPKAL23B6EOLQCLSWBGHXHEALBDZU/ > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/4QBBBQI7ZHW7IYYU57FXXBIOWM25NTCN/
[Distutils] Re: packaging guide and private packages index
There’s also pypiserver - https://github.com/pypiserver/pypiserver It’s a simple package server. Nowhere near as fully featured as devpi, but super simple to get up and running. I am a maintainer and have used it in production settings. Feel free to let me know if you have questions On Mon, Jul 30, 2018 at 5:02 PM David Cournapeau wrote: > Hi, > > I see that there is almost no mention of private packages index in the > packaging guide, and no recommendation on what to use. > > Currently googling for private packages mostly return obsolete (and not > very practical) recommendations based on dependency links. > > In 2018, what would be the recommended practices for teams that develop > private packages that depend on each other ? > > thank you, > > David > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/QLDBOAJZOZACT3FGHFC664DLHOHMBLDT/ > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/FRFLRUDLT4WOEKZDRSFHY3MMCWYSCQSL/
[Distutils] Re: packaging guide and private packages index
When I was in that situation I just reverse engineered a very *very* simple PyPi clone: https://github.com/waynew/flaskpypi There aren't actually any docs, but you just have to put your packages in a specific directory and it will serve them up for you. You just have to tell pip to use the url for that index. If it doesn't have the package/version locally then it will check the proper pypi for it. HTH, Wayne === I welcome VSRE emails. Learn more at http://vsre.info/ === On Mon, Jul 30, 2018 at 5:04 PM David Cournapeau wrote: > Hi, > > I see that there is almost no mention of private packages index in the > packaging guide, and no recommendation on what to use. > > Currently googling for private packages mostly return obsolete (and not > very practical) recommendations based on dependency links. > > In 2018, what would be the recommended practices for teams that develop > private packages that depend on each other ? > > thank you, > > David > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/QLDBOAJZOZACT3FGHFC664DLHOHMBLDT/ > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/7YMKPKAL23B6EOLQCLSWBGHXHEALBDZU/
[Distutils] Re: packaging guide and private packages index
On 30 July 2018 at 23:01, David Cournapeau wrote: > Hi, > > I see that there is almost no mention of private packages index in the > packaging guide, and no recommendation on what to use. > > Currently googling for private packages mostly return obsolete (and not very > practical) recommendations based on dependency links. > > In 2018, what would be the recommended practices for teams that develop > private packages that depend on each other ? That's a very good point, and I think it would be a really useful addition to the packaging guide. I believe the most commonly recommended approach is to use a local,devpi instance as a private index, but in terms of development workflow around that basis, I don't really know that there's much in the way of a consensus. (I should say that I don't work in an environment like this myself, so my comments are based purely on what I've heard, not on personal experience). Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/QVRQV2MILNRNWZOUXLHTTSS6U5HY3WTA/
