Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Nick Coghlan]

On 22 August 2016 at 02:38, Thomas Kluyver  wrote:
> I think shelling out is a reasonable approach for certain operations if the
> tool you're shelling out to is designed for it. When we write command-line
> programs, we often think about users typing in commands at a terminal, and
> design things that are flexible and somewhat smart. If shelling out is to be
> an API, the commands need to be predictable and dumb.

And even then providing a decent error reporting API can be quite
difficult compared to the in-Python error reporting options with full
exception support.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Thomas Kluyver]

On Sun, Aug 21, 2016, at 05:19 PM, Jim Fulton wrote:
> I'm open to shelling out, but pessimistic that it will turn out well.
> I started with that approach initially with easy_install and it fell
> apart quickly.  But when we get into it... who knows?

I think shelling out is a reasonable approach for certain operations
if the tool you're shelling out to is designed for it. When we write
command-line programs, we often think about users typing in commands
at a terminal, and design things that are flexible and somewhat
smart. If shelling out is to be an API, the commands need to be
predictable and dumb.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Jim Fulton]

On Sun, Aug 21, 2016 at 12:29 PM, Donald Stufft  wrote:

>
> On Aug 21, 2016, at 12:19 PM, Jim Fulton  wrote:
>
> I'm open to shelling out, but pessimistic that it will turn out well. I
> started with that approach initially with easy_install and it fell apart
> quickly.  But when we get into it... who knows?
>
>
>
> Shelling out is currently the only exposed “API” that pip has, we’re not
> opposed to adding extra APIs though. Our current approach has been to wait
> and see for people to come out with specific use cases they have for an API
> and then work together to figure out what API we can create that satisfies
> that. Thus far we’ve accomplished this by creating new libraries that
> aren’t pip and moving functionality out of pip (and setuptools) and into
> those libraries, and then making pip/setuptools consume those. This has
> generally worked pretty well I think, as it’s easier to be careful not to
> accidentally expose some terrible internal details of pip as public API
> when it’s a new, carefully designed thing, and we can make working with
> those libraries better than it is to simply expose some part of pip. We
> generally pair this along with defining things in PEPs so that these new
> libraries don’t become the new distutils/setuptools/pip (e.g.,
> implementation defined standards) which should ideally allow anyone to
> create a from scratch implementation and have it interopt just fine.
>

Sounds reasonable. (I'd seen similar statements before, which I'd alluded
to in another message.) Thanks.

Jim

-- 
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Donald Stufft]

> On Aug 21, 2016, at 12:19 PM, Jim Fulton  wrote:
> 
> I'm open to shelling out, but pessimistic that it will turn out well. I 
> started with that approach initially with easy_install and it fell apart 
> quickly.  But when we get into it... who knows?


Shelling out is currently the only exposed “API” that pip has, we’re not 
opposed to adding extra APIs though. Our current approach has been to wait and 
see for people to come out with specific use cases they have for an API and 
then work together to figure out what API we can create that satisfies that. 
Thus far we’ve accomplished this by creating new libraries that aren’t pip and 
moving functionality out of pip (and setuptools) and into those libraries, and 
then making pip/setuptools consume those. This has generally worked pretty well 
I think, as it’s easier to be careful not to accidentally expose some terrible 
internal details of pip as public API when it’s a new, carefully designed 
thing, and we can make working with those libraries better than it is to simply 
expose some part of pip. We generally pair this along with defining things in 
PEPs so that these new libraries don’t become the new distutils/setuptools/pip 
(e.g., implementation defined standards) which should ideally allow anyone to 
create a from scratch implementation and have it interopt just fine.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Jim Fulton]

On Sat, Aug 20, 2016 at 6:31 PM, Wes Turner  wrote:
...

> I may be optimistic, but should adding buildout support for wheel be more
> complicated than shelling out to pip with the correct, uhm, --prefix etc?
>

I'm open to shelling out, but pessimistic that it will turn out well. I
started with that approach initially with easy_install and it fell apart
quickly.  But when we get into it... who knows?

Jim

-- 
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Jim Fulton]

On Sat, Aug 20, 2016 at 11:07 PM, Nick Coghlan  wrote:

> On 21 August 2016 at 05:46, Jim Fulton  wrote:
> > On Sat, Aug 20, 2016 at 3:02 PM, Nick Coghlan 
> wrote:
> >> > I have the impression that uninstalling things can be
> >> > problematic, but maybe that's been fixed.
> >>
> >> Uninstallation is fine, as we *do* have a full file manifest after a
> >> component has been installed.
> >
> > So, if package A and B install the same file, X, and then A is
> uninstalled,
> > is X still there?  If then B is uninstalled, does X go away?  Does this
> > machinery depend on whether X has the the same contents in A and B?
>
> If A and B install to the same path, B will overwrite A's version, and
> if either is uninstalled, they'll delete whichever version is
> currently there. Since "the same path" in this case usually refers to
> either a script or a Python module, the unhandled conflict is really
> at install time - we'd want something closer to the RPM/deb style "A
> file already exists at that destination, so we're not installing the
> requested package at all", rather than a Windows-style reference
> counting system.
>
> As Daniel notes, this kind of check is actually already possible when
> installing from a wheel file today, it just requires someone with the
> roundtuits to add it to pip (perhaps via a new vendorable package for
> working with Python installation metadata and detecting conflicts
> between what's already installed and a wheel being considered for
> installation).
>

The most common source of conflicts, I expect, are namespace package
__init__.py files. You wouldn't want to reject installing a package with a
conflicting __init__.py file in a namespace package.  I realize that PEP420
and the demise of Python 2 should make this case go away, someday, although
PEP420 doesn't prohibit __init__.py files in namespace packages.

OK, I'm done beating this horse. It was dead a while ago. :)

Jim

-- 
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Nathaniel Smith]

On Sun, Aug 21, 2016 at 3:51 AM, Paul Moore  wrote:
> On 20 August 2016 at 22:31, Nathaniel Smith  wrote:
>> Wheels are a pretty simple and straightforward format. They've got some
>> metadata, and then the are a set of directories with labels attached: "this
>> directory needs to go on sys.path", "this directory has scripts that should
>> have shebang fixups applied and then go on $PATH", etc. (The spec is not
>> long: https://www.python.org/dev/peps/pep-0427/)
>>
>> My first guess at how buildout could handle wheels elegantly would be:
>>
>> - we have one piece of code that abstracts the operation of "please give me
>> a wheel for package X version Y that works on this system".
>
> This can be abstracted at the moment as "pip wheel ".
> That will deliver a wheel to the current directory which is exactly
> what is described above.

Or possibly "pip wheel --no-deps " or something similar.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Paul Moore]

On 20 August 2016 at 22:31, Nathaniel Smith  wrote:
> Wheels are a pretty simple and straightforward format. They've got some
> metadata, and then the are a set of directories with labels attached: "this
> directory needs to go on sys.path", "this directory has scripts that should
> have shebang fixups applied and then go on $PATH", etc. (The spec is not
> long: https://www.python.org/dev/peps/pep-0427/)
>
> My first guess at how buildout could handle wheels elegantly would be:
>
> - we have one piece of code that abstracts the operation of "please give me
> a wheel for package X version Y that works on this system".

This can be abstracted at the moment as "pip wheel ".
That will deliver a wheel to the current directory which is exactly
what is described above. It probably doesn't meet buildout's
repeatability requirements (as it could invoke a build step if there's
no published wheel, and build steps can do anything they like -
--only-binary would be a way to avoid that, at the cost of failing if
a project didn't publish a wheel) but it would likely work as a
prototype.

I don't think there's anything inherent in buildout's isolation model
of using sys.path entries for each dependency, that is incompatible
with pip/wheels. If buildout unpacks eggs, it can do the same with
wheels. If it uses zipped eggs, then it can *probably* do the same
with wheels, although there's no "zip safe" mechanism to declare a
wheel as safe for this use.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Nick Coghlan]

On 21 August 2016 at 05:46, Jim Fulton  wrote:
> On Sat, Aug 20, 2016 at 3:02 PM, Nick Coghlan  wrote:
>> > I have the impression that uninstalling things can be
>> > problematic, but maybe that's been fixed.
>>
>> Uninstallation is fine, as we *do* have a full file manifest after a
>> component has been installed.
>
> So, if package A and B install the same file, X, and then A is uninstalled,
> is X still there?  If then B is uninstalled, does X go away?  Does this
> machinery depend on whether X has the the same contents in A and B?

If A and B install to the same path, B will overwrite A's version, and
if either is uninstalled, they'll delete whichever version is
currently there. Since "the same path" in this case usually refers to
either a script or a Python module, the unhandled conflict is really
at install time - we'd want something closer to the RPM/deb style "A
file already exists at that destination, so we're not installing the
requested package at all", rather than a Windows-style reference
counting system.

As Daniel notes, this kind of check is actually already possible when
installing from a wheel file today, it just requires someone with the
roundtuits to add it to pip (perhaps via a new vendorable package for
working with Python installation metadata and detecting conflicts
between what's already installed and a wheel being considered for
installation).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Wes Turner]

On Saturday, August 20, 2016, Nathaniel Smith  wrote:

> On Aug 20, 2016 11:00 AM, "Jim Fulton"  > wrote:
> >
> [...]
> > With buildout, I chose to use eggs differently.  I simply generate
> scripts with dependencies explicitly listed in the Python path.  This is
> very easy to reason about.  It makes it easy to see what's installed by
> looking at the generated path.  This is similar to the way Java uses class
> paths to assemble jar files, which is fitting given that the design of eggs
> took at least some inspiration from jar files.
> >
> > I'm not a fan of Java, but I think class paths + jar files was something
> it got right.  Each program has it's own distinct environment.  If you need
> to add/remove/update a dependency, just change the path.  Want to share
> installed components between programs?  It's easy because only one
> installed component needs to be stored and different programs point to it
> via their paths.
>
> Wheels are a pretty simple and straightforward format. They've got some
> metadata, and then the are a set of directories with labels attached: "this
> directory needs to go on sys.path", "this directory has scripts that should
> have shebang fixups applied and then go on $PATH", etc. (The spec is not
> long: https://www.python.org/dev/peps/pep-0427/)
>
> My first guess at how buildout could handle wheels elegantly would be:
>
> - we have one piece of code that abstracts the operation of "please give
> me a wheel for package X version Y that works on this system". This then
> contacts pypi, builds sdists if it has to, whatever. It could be in the
> form of an invocation of pip, or it could be a call into some common
> library that gets factored out of pip and used by both pip and buildout,
> whatever.
>
> - pip uses this code to get a wheel, and once it has a wheel, it installs
> it into an environment in the same way it does today.
>
> - buildout *also* uses this code to get a wheel, and once it has a wheel,
> it does its own thing: maybe its own dependency resolution, and definitely
> its own install strategy, where it unpacks the wheel's directories into its
> own preferred layout, does sys.path munging on the wheel's scripts, etc.
>
> AFAICT there's nothing in the wheel format that ties you to the
> virtualenv-style directory layout.
>
> The disadvantage is that someone would have to do the work to factor out
> that get-a-wheel code, and to write/maintain custom wheel installation code
> in buildout, instead of just invoking setuptools. The advantage would be
> the now you aren't dependent on setuptools...
>
> > The buildout developers have discussed options for the future. We know
> there's a reckoning coming, but so far, thankfully, we've been able to put
> it off, but we don't want to be a burden on the rest of the Python
> community. (Seriously, thank you for not breaking us. :) )
>

I may be optimistic, but should adding buildout support for wheel be more
complicated than shelling out to pip with the correct, uhm, --prefix etc?

... recipes for buildout & pip, omelette symlinks:
http://www.mail-archive.com/distutils-sig@python.org/msg24964.html
Re: [Distutils] Future of setuptools and buildout


Re: buildout and pip and wheel

"Add support for installing
wheels"https://github.com/buildout/buildout/issues/144






> As a heads up, there is unfortunately another source of breakage heading
> down the pipe: the PEP 516/517 plans for making it easy to use
> non-distutils-based build systems are heavily tied to wheels, so if/when
> the spec gets finished and projects start adopting alternative build
> systems, you may see an increasing proliferation of projects that cannot
> build eggs or be easy_install'ed.
>
> (Ref: https://www.python.org/dev/peps/pep-0517/)
>
> It's not clear when this will happen exactly -- I'm in the midst of
> dealing with some medical stuff, so haven't had much/any time for
> distutils-sig stuff recently, but sooner or later this is the direction we
> want to go.
>
> -n
>

So these will all still install with pip?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Wes Turner]

On Saturday, August 20, 2016, Daniel Holth  wrote:

> We have manifests before and after in wheel but good conflict management
> is still theoretical. I think someone had a prototype years ago. For
> example checking checksums on uninstall and maybe not uninstalling modified
> [configuration] files.
>
- Uninstalling X would break Y and Z
- Because there are no additional transitive dependencies, uninstalling X
should also uninstall Y and Z
- ./etc/x.cfg is modified, don't remove it on_uninstall



>
> On Sat, Aug 20, 2016, 15:47 Jim Fulton  > wrote:
>
>>
>>
>> On Sat, Aug 20, 2016 at 3:02 PM, Nick Coghlan > > wrote:
>>
>> ...
>>
>> > I have the impression that uninstalling things can be
>>>
>> > problematic, but maybe that's been fixed.
>>>
>>> Uninstallation is fine, as we *do* have a full file manifest after a
>>> component has been installed.
>>>
>>
>> So, if package A and B install the same file, X, and then A is
>> uninstalled, is X still there?  If then B is uninstalled, does X go away?
>> Does this machinery depend on whether X has the the same contents in A and
>> B?
>>
>> ...
>>
>> Jim
>>
>> --
>> Jim Fulton
>> http://jimfulton.info
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> 
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Nathaniel Smith]

On Aug 20, 2016 11:00 AM, "Jim Fulton"  wrote:
>
[...]
> With buildout, I chose to use eggs differently.  I simply generate
scripts with dependencies explicitly listed in the Python path.  This is
very easy to reason about.  It makes it easy to see what's installed by
looking at the generated path.  This is similar to the way Java uses class
paths to assemble jar files, which is fitting given that the design of eggs
took at least some inspiration from jar files.
>
> I'm not a fan of Java, but I think class paths + jar files was something
it got right.  Each program has it's own distinct environment.  If you need
to add/remove/update a dependency, just change the path.  Want to share
installed components between programs?  It's easy because only one
installed component needs to be stored and different programs point to it
via their paths.

Wheels are a pretty simple and straightforward format. They've got some
metadata, and then the are a set of directories with labels attached: "this
directory needs to go on sys.path", "this directory has scripts that should
have shebang fixups applied and then go on $PATH", etc. (The spec is not
long: https://www.python.org/dev/peps/pep-0427/)

My first guess at how buildout could handle wheels elegantly would be:

- we have one piece of code that abstracts the operation of "please give me
a wheel for package X version Y that works on this system". This then
contacts pypi, builds sdists if it has to, whatever. It could be in the
form of an invocation of pip, or it could be a call into some common
library that gets factored out of pip and used by both pip and buildout,
whatever.

- pip uses this code to get a wheel, and once it has a wheel, it installs
it into an environment in the same way it does today.

- buildout *also* uses this code to get a wheel, and once it has a wheel,
it does its own thing: maybe its own dependency resolution, and definitely
its own install strategy, where it unpacks the wheel's directories into its
own preferred layout, does sys.path munging on the wheel's scripts, etc.

AFAICT there's nothing in the wheel format that ties you to the
virtualenv-style directory layout.

The disadvantage is that someone would have to do the work to factor out
that get-a-wheel code, and to write/maintain custom wheel installation code
in buildout, instead of just invoking setuptools. The advantage would be
the now you aren't dependent on setuptools...

> The buildout developers have discussed options for the future. We know
there's a reckoning coming, but so far, thankfully, we've been able to put
it off, but we don't want to be a burden on the rest of the Python
community. (Seriously, thank you for not breaking us. :) )

As a heads up, there is unfortunately another source of breakage heading
down the pipe: the PEP 516/517 plans for making it easy to use
non-distutils-based build systems are heavily tied to wheels, so if/when
the spec gets finished and projects start adopting alternative build
systems, you may see an increasing proliferation of projects that cannot
build eggs or be easy_install'ed.

(Ref: https://www.python.org/dev/peps/pep-0517/)

It's not clear when this will happen exactly -- I'm in the midst of dealing
with some medical stuff, so haven't had much/any time for distutils-sig
stuff recently, but sooner or later this is the direction we want to go.

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Daniel Holth]

We have manifests before and after in wheel but good conflict management is
still theoretical. I think someone had a prototype years ago. For example
checking checksums on uninstall and maybe not uninstalling modified
[configuration] files.

On Sat, Aug 20, 2016, 15:47 Jim Fulton  wrote:

>
>
> On Sat, Aug 20, 2016 at 3:02 PM, Nick Coghlan  wrote:
>
> ...
>
> > I have the impression that uninstalling things can be
>>
> > problematic, but maybe that's been fixed.
>>
>> Uninstallation is fine, as we *do* have a full file manifest after a
>> component has been installed.
>>
>
> So, if package A and B install the same file, X, and then A is
> uninstalled, is X still there?  If then B is uninstalled, does X go away?
> Does this machinery depend on whether X has the the same contents in A and
> B?
>
> ...
>
> Jim
>
> --
> Jim Fulton
> http://jimfulton.info
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Jim Fulton]

On Sat, Aug 20, 2016 at 3:02 PM, Nick Coghlan  wrote:

...

> I have the impression that uninstalling things can be
>
> problematic, but maybe that's been fixed.
>
> Uninstallation is fine, as we *do* have a full file manifest after a
> component has been installed.
>

So, if package A and B install the same file, X, and then A is uninstalled,
is X still there?  If then B is uninstalled, does X go away?  Does this
machinery depend on whether X has the the same contents in A and B?

...

Jim

-- 
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Nick Coghlan]

On 21 August 2016 at 04:00, Jim Fulton  wrote:
> These really weren't goals of setuptools, which tried to fit into
> site-package-based installs and ironically resorted to unsavory techniques
> to wedge eggs in using elaborate .pth files.

Right, and pip took that approach further by making the site-packages
friendly approach the *default* approach, rather than a selectable
option.

This has had the effect of making pip not only useful for component
management in its own right, but also viable as a tool for assembling
*downstream* packages from Python source projects.

> With buildout, I chose to use eggs differently.  I simply generate scripts
> with dependencies explicitly listed in the Python path.  This is very easy
> to reason about.  It makes it easy to see what's installed by looking at the
> generated path.  This is similar to the way Java uses class paths to
> assemble jar files, which is fitting given that the design of eggs took at
> least some inspiration from jar files.
>
> I'm not a fan of Java, but I think class paths + jar files was something it
> got right.  Each program has it's own distinct environment.  If you need to
> add/remove/update a dependency, just change the path.  Want to share
> installed components between programs?  It's easy because only one installed
> component needs to be stored and different programs point to it via their
> paths.

Agreed, but similar to both conda and JARs themselves, this improved
isolation made buildout *less* useful to folks working on distro
packages that actually *wanted* to be installing their Python
components into the system Python installation.

> Pip's approach OTOH, leaves me skeptical.  When installing a package, it has
> to unpack it and install its files into a shared directory structure where
> conflicts are likely.  How does it handle these conflicts? IDK. I should,
> but I don't.

It doesn't, since we don't have full file manifests in our metadata -
if you inadvertently install both python-openid and python3-openid
into the same virtualenv, they'll trample over each other's files.

> I have the impression that uninstalling things can be
> problematic, but maybe that's been fixed.

Uninstallation is fine, as we *do* have a full file manifest after a
component has been installed.

> At best, this is a lot of
> complexity to maintain, at worse, uninstalls/reinstalls leave things behind
> that make buildout's goal of repeatability impossible to achieve.
>
> For isolation, pip relies on virtualenv.  This has always struck me as an
> extremely heavy-handed approach. I'm quite surprised that the Python
> community settled on it.  But whatever.

The pay-off for the pip model comes in the fact that using venv is
*optional* in a way that isn't generally true for other more
specifically app focused systems like conda and buildout. That doesn't
make conda and buildout wrong - it means they cut out a particular
rare-in-number-but-large-in-influence use case (Linux based operating
system development) in order to better focus on a more targeted set of
use cases (data analysis for conda, network service development for
buildout)

> A few packages (e.g. NumPy) really
> depend on the shape of the environment they're installed into so can't be
> installed with buildout, but can be installed with pip+virtualenv.

The distinction also goes the other way - pip can be used to add
capabilities to an existing Python installation without fundamentally
changing the architecture of that installation, including how it
decides to handle (or not handle, as the case may be) separating
different applications from each other.

The key thing that pip's downstream platform tend to bring to the
table is different answers to that application isolation problem,
while pip just handles dependency management and build system
invocation:

- virtualenv mainly just manipulates sys.path to amend where
site-packages is found
- Linux distros offer their system package management (including full
preinstall file manifests and associated conflict detection) as well
as chroots and Linux containers (most famously, Docker)
- *nix systems have also long offered the "modules" environment
management utility (especially popular in HPC)
- buildout has tailored per-application sys.path definitions
- conda has its own environment management tooling

> The buildout developers have discussed options for the future. We know
> there's a reckoning coming, but so far, thankfully, we've been able to put
> it off, but we don't want to be a burden on the rest of the Python
> community. (Seriously, thank you for not breaking us. :) )

I don't think buildout's a burden. While there's no logical reason for
environment isolation to be tightly coupled to dependency management
and build system invocation, when designing buildout you didn't have a
choice - setuptools was realistically the only game in town for those
two pieces, so it made sense to structure buildout around that, and
then 

[Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Jim Fulton]

Buildout was designed to automate project development and deployment tasks.
This encompasses assembly of software packages (Python, JavaScript, etc.)
and generation of support artifacts such as configuration files and scripts.

I never set out to make buildout a Python package installer.

I adopted setuptools because it enabled most of what I needed in terms of
fetching and assembling Python packages.  Adopting eggs was a by product of
adopting setuptools.

Some basic goals of buildout:


   - Isolation from system-provided Python installations.  This was a
   result of problems experienced when deploying applications for customers
   whos Python environments were different from ours (Zope Corporation's).

   (Thankfully, Docker mitigates these issues quite a bit.)


   - Repeatability -- Given the same buildout configuration file, running
   buildouts in separate workspaces should product the same result for a given
   platform, regardless of their history.

   One consequence of this is that it's important to be able uninstall
   things cleanly, both artifacts generated by buildout and packages installed
   by it.

   - Tight control over program dependencies.  This means that different
   programs (scripts) in a buildout could depend on different versions and
   maybe more importantly, dependencies didn't leak from one program to
   another.

   Expanding on this last point, it's common for libraries to change their
   behavior depending on whether another library is importable.  With a shared
   set of packages, installing a new program could cause introduction of
   packages that change the behavior of existing programs.  This would be bad.


These really weren't goals of setuptools, which tried to fit into
site-package-based installs and ironically resorted to unsavory techniques
to wedge eggs in using elaborate .pth files.

With buildout, I chose to use eggs differently.  I simply generate scripts
with dependencies explicitly listed in the Python path.  This is very easy
to reason about.  It makes it easy to see what's installed by looking at
the generated path.  This is similar to the way Java uses class paths to
assemble jar files, which is fitting given that the design of eggs took at
least some inspiration from jar files.

I'm not a fan of Java, but I think class paths + jar files was something it
got right.  Each program has it's own distinct environment.  If you need to
add/remove/update a dependency, just change the path.  Want to share
installed components between programs?  It's easy because only one
installed component needs to be stored and different programs point to it
via their paths.

Pip's approach OTOH, leaves me skeptical.  When installing a package, it
has to unpack it and install its files into a shared directory structure
where conflicts are likely.  How does it handle these conflicts? IDK. I
should, but I don't.  I have the impression that uninstalling things can be
problematic, but maybe that's been fixed. At best, this is a lot of
complexity to maintain, at worse, uninstalls/reinstalls leave things behind
that make buildout's goal of repeatability impossible to achieve.

For isolation, pip relies on virtualenv.  This has always struck me as an
extremely heavy-handed approach. I'm quite surprised that the Python
community settled on it.  But whatever. A few packages (e.g. NumPy) really
depend on the shape of the environment they're installed into so can't be
installed with buildout, but can be installed with pip+virtualenv.

The buildout developers have discussed options for the future. We know
there's a reckoning coming, but so far, thankfully, we've been able to put
it off, but we don't want to be a burden on the rest of the Python
community. (Seriously, thank you for not breaking us. :) )

We've debated just invoking virtualenv and pip to assemble Python
applications.  A model we've been discussing is to let buildout recipes do
this.  No change is needed to buildout.  There's at least one and probably
multiple recipes that do this, although I haven't used them myself. In this
model, a buildout could install different virtualenvs for different
programs, allowing dependencies to be kept distinct.  I still worry about
the integrity of these virtualenvs over time as dependencies are added,
removed, and updated.

If I could have my way, the path of distinct package directories approach
would still be an option for buildout, as I think it's superior.  I'm
hopeful that it will be possible to use wheels this way, assuming that eggs
are retired.

I would also prefer that there be one library, or set of complementary
libraries, to find, download and install packages.  I normally like
competition, but this is such a boring and difficult domain that I don't
really see there being interest in developing multiple solutions. Honestly,
I'd be nervous if, in the long run, buildout and pip installed things
differently, especially given security considerations.

In the long run, I suspect it