Re: [Distutils] Future of setuptools and buildout

2016-08-20 Thread David Cournapeau 
On Sat, Aug 20, 2016 at 6:30 PM, Nick Coghlan  wrote:

> On 20 August 2016 at 02:46, Chris Barker  wrote:
> > Maybe the alternative to setuptools_lite is to set up one of these other
> > tools to work well (easily) for building python packages. If it can do
> > everything that setuptools can do (that we want setuptools to do), and
> just
> > as easily, maybe that's the next step forward.
> >
> > but if that isn't happening soon, then a setuptools_lite would be a
> useful
> > step forward.
>
> Enabling folks to more easily use existing build systems like Scons,
> Waf, Meson, CMake, et al is one of the main goals of pyproject.toml.
>
> Daniel Holth has a working Scons proof of concept at
> https://pypi.python.org/pypi/enscons that actually works independently
> of pyproject.toml support in pip by using a boilerplate setup.py to do
> the bootstrapping and invocation of the build system.
>
> It's why we don't particularly want to replace distutils/setuptools
> with any build system in particular - the world actually has a surfeit
> of cross-platform build systems, and many of them are written in
> Python and are hence quite friendly to being bootstrapped for use in
> building Python extension modules.
>

Indeed. Having been involved w/ the mess that is numpy.distutils, this was
(and still is) the major issue w/ distutils for the scientific/pydata crowd.

Conflating packaging and building is an impediment to deploy software to
different platforms (build is generally better handled by developers,
packaging by platform-specific packagers).

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

2016-08-20 Thread Nick Coghlan 
On 20 August 2016 at 02:46, Chris Barker  wrote:
> Maybe the alternative to setuptools_lite is to set up one of these other
> tools to work well (easily) for building python packages. If it can do
> everything that setuptools can do (that we want setuptools to do), and just
> as easily, maybe that's the next step forward.
>
> but if that isn't happening soon, then a setuptools_lite would be a useful
> step forward.

Enabling folks to more easily use existing build systems like Scons,
Waf, Meson, CMake, et al is one of the main goals of pyproject.toml.

Daniel Holth has a working Scons proof of concept at
https://pypi.python.org/pypi/enscons that actually works independently
of pyproject.toml support in pip by using a boilerplate setup.py to do
the bootstrapping and invocation of the build system.

It's why we don't particularly want to replace distutils/setuptools
with any build system in particular - the world actually has a surfeit
of cross-platform build systems, and many of them are written in
Python and are hence quite friendly to being bootstrapped for use in
building Python extension modules.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

2016-08-19 Thread Chris Barker 
On Thu, Aug 18, 2016 at 10:17 PM, Wes Turner  wrote:

> setuptools does all of these, yes? but think of these in terms of when
>> they come into play:
>>
>> build time:
>>- building a package
>>
>
> - building c extensions
> - building NumPy (fortran,)
>

exactly! which is WHY we want/need more flexibility in build tools --
distutils is a mess to extend to do other things. (though is does do a fine
job with pure C extensions, which is what it was designed for).



> install time: installing a package;
>>   - installing a package
>>   - dependency management
>>
> run time
>>   - run-time version management
>>   - plugin handling, etc.
>>
>
> - running tests
>   - (It's much faster/easier/consistent/reproducible to run tests when
> all of the dependencies are bundled into one executable ZIP (e.g. PEX))
>

hmm - it seems most of us use a test-running for this -- so I put that in a
separate category. But you need the package management tool to be able to
easily set up an environment for testing -- i.e install the package and all
its dependencies, then run the tests. I don't know anything about PEX, but
I'd really NOT want the test runner to rely on any particular installation
setup.

And, as it happend, test running is one thing setuptools does not
currently, do so, we have pytest and nose and ??? that actually ARE
separate pluggable test running systems.


>
>> so we have:
>>
>>  - The mythical build tool
>>
>
> - TBH, I don't know much about pyproject.toml
>   - "PEP 518 -- Specifying Minimum Build System Requirements for Python
> Projects"
>  https://www.python.org/dev/peps/pep-0518/
>

pyproject.toml is NOT a build tool at all -- but it is a step towards being
able to support other build and packaging tools.

so maybe version 2 of setuptools_lite should rely on pyproject.toml.


> And then there are a number of build tools which work with multiple
> languages (because just solving for Python doesn't get it):
>
> - Conda (Python)
>

conda IS NOT a build tool -- I was pretty disappointed when I realized
that, but what can you do? cross-platform, cross-language build tools are a
really hard problem -- no one wants to write another one :-). And this is a
good thing -- what we WANT is for the packaging tool and the build tools to
be separate.


>   - meta.yaml, build.sh, build.bat *
>

the meta.yaml provides all the meta-data (kind like a higher level
pyproject.toml, and then simply invokes the build scripts. but you need to
write those build scripts -- conda provides literally no help with that.
most of them simply invoke whatever build system the library you are trying
to [package already uses -- setuptools, make, cmake, 

However, since you brought up testing earlier -- conda does have a nice
system for testing -- it isn't a test runner, but it does provide isolated
environments (a job of the installer???) -- so conda-build can:

setup an environment for building
build the package
Then:

setup an environment for testing
installed the package that was just built
invoke a test-runner

This is REALLY nice -- you can have all your test code run with the built
package, with the environment that you have specified, on the platform you
have built for (on).

This has caught all sorts of stupid errors for me -- forgetting a
dependency, etc...

Whether that should be built into the packaging tool I suppose is
debateable. But maybe it's not -- conda-build is a separate tool -- it
relies on the conda system, but it does it's own thing. conda packages are
fairly simple archives -- it's very possible to create them with other
tools.

- Blaze (Java, Python?; Google) -> Open Sourced as Bazel
>
- Pants (Python; Twitter):
> - Buck (Java, Facebook):
> - Bazel (Java, Python; Google)
>

Maybe the alternative to setuptools_lite is to set up one of these other
tools to work well (easily) for building python packages. If it can do
everything that setuptools can do (that we want setuptools to do), and just
as easily, maybe that's the next step forward.

but if that isn't happening soon, then a setuptools_lite would be a useful
step forward.

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

2016-08-18 Thread Wes Turner 
On Thu, Aug 18, 2016 at 4:54 PM, Chris Barker  wrote:

> On Wed, Aug 17, 2016 at 6:45 PM, Daniel Holth  wrote:
>
>> And a while back I argued against setuptools-lite, because I thought it
>> would not solve the poor extensibility problem that stems from its basic
>> distutils derived design... which includes all the classes and subclasses
>> that have to work together to do its thing. My own thinking is that
>> setuptools is fine, except that when you need to extend it you have a big
>> problem; so make it easy to replace.
>>
>
> I never had any fantasies that setuptool-lite would improve on this in a
> way shape or form -- indeed, a better build system is likely to need to be
> built from scratch, or on a totally different framework, anyway.
>
>
>> Does pip monkey patch setuptools? I thought it just forced setuptools
>> even if a package only asked for distutils.
>>
>
> good point -- not really a monkey patch, but it does force particular
> configuration.
>
>
>> So you are suggesting setuptools_lite would not honor the setup_requires
>> = [] and test_requires = []
>>
>
> essentially, yes. Though It would be nice if that data could be available
> for pip -- and I"m not sure how that is done at this point.
>
>
>> and perhaps "setup.py install" would also throw an error?
>>
>
> Probably not -- but it could only install from local source -- like
> distutils. Though if we could re-work it to only do develop mode and build
> wheels, that may be good -- you'd need pip to install it.
>
>
>> What else? I would suggest that to effectively argue the idea requires
>> reading the source code and enumerating exactly which complexity goes away
>> with the _lite version.
>>
>
> I don't know that reading the source is the issue -- I'm thinking entirely
> in terms of stripping out the API, so there may be things that need to stay
> in, but we don't expose them. though maybe even that's harder than I think.
>
> But yes -- enumerating what would and wouldn't be supported would be a
> good start. But here is the guideline:
>
> It seems we have a (maybe vague) vision of what we want the tool ecosystem
> to look like that involves separation of concerns, such that each component
> is replaceable without having to change the others:
>
> The concerns:
>   - building a package
>   - installing a package
>   - dependency management
>   - run-time version management
>   - plugin handling, etc.
>   - resource managment
>
> setuptools does all of these, yes? but think of these in terms of when
> they come into play:
>
> build time:
>- building a package
>

- building c extensions
- building NumPy (fortran,)


> install time: installing a package;
>   - installing a package
>   - dependency management
>
run time
>   - run-time version management
>   - plugin handling, etc.
>

- running tests
  - (It's much faster/easier/consistent/reproducible to run tests when all
of the dependencies are bundled into one executable ZIP (e.g. PEX))


>
> so we have:
>
>  - The mythical build tool
>

- TBH, I don't know much about pyproject.toml
  - "PEP 518 -- Specifying Minimum Build System Requirements for Python
Projects"
 https://www.python.org/dev/peps/pep-0518/

And then there are a number of build tools which work with multiple
languages (because just solving for Python doesn't get it):

- Conda (Python)
  - meta.yaml, build.sh, build.bat *
  - http://conda.pydata.org/docs/build_tutorials.html
  -
http://conda.pydata.org/docs/build_tutorials/pkgs.html#conda-build-skeleton
(pypkg -> conda)
  - http://conda.pydata.org/docs/building/meta-yaml.html#build-section
  - http://conda.pydata.org/docs/bdist_conda.html (condapkg -> pypkg)

- Blaze (Java, Python?; Google) -> Open Sourced as Bazel

- Pants (Python; Twitter):
  - BUILD files
  - https://pantsbuild.github.io/python_readme.html
  - https://pantsbuild.github.io/3rdparty_py.html (pip requirements.txt)
  - https://pantsbuild.github.io/scala.html
  - https://pantsbuild.github.io/go_readme.html

- Buck (Java, Facebook):
  - https://buckbuild.com/rule/prebuilt_python_library.html (.egg)
  - https://buckbuild.com/rule/python_binary.html
  - https://buckbuild.com/rule/python_library.html
  - https://buckbuild.com/rule/python_test.html
  - https://buckbuild.com/rule/cxx_library.html
  - https://buckbuild.com/rule/go_library.html
  - https://buckbuild.com/rule/rust_library.html

- Bazel (Java, Python; Google)
  - http://bazel.io/docs/be/python.html
 - http://bazel.io/docs/be/python.html#py_binary
 - http://bazel.io/docs/be/python.html#py_library
 - http://bazel.io/docs/be/python.html#py_test
  - https://bazel.io/docs/be/c-cpp.html
  - https://github.com/bazelbuild/rules_go


... PEX is based on PEP 441:

- "PEP 441 -- Improving Python ZIP Application Support"
  https://www.python.org/dev/peps/pep-0441/
- https://github.com/pantsbuild/pex/blob/master/docs/whatispex.rst

- Again, wheel includes a file manifest in the .whl .zip with SHA256

Re: [Distutils] Future of setuptools and buildout

2016-08-18 Thread Wes Turner 
Re: buildout and pip and wheel

"Add support for installing wheels"
https://github.com/buildout/buildout/issues/144

It's been awhile since I've worked with buildout (for Zope 2, Plone,
AppEngine zipimports).

This reads #egg= links from pip requirements files:

-
https://github.com/collective/collective.recipe.pip/blob/master/collective/recipe/pip/__init__.py

This installs packages with buildout and pip (instead of zc.recipe.egg)
into the eggs/ directory. IDK if it supports wheels? (it specifies --egg):

-
https://github.com/k4ml/mk.recipe.pip/blob/master/mk/recipe/pip/__init__.py

... this builds a usable merged namespace with symlinks to eggs:

- https://github.com/collective/collective.recipe.omelette/


On Tuesday, August 16, 2016, Daniel Holth  wrote:

> There are two proposals to add pluggable build systems to Python sdists,
> and one of them will probably be implemented. You add a pyproject.toml to
> the root of your sdist, which the installer uses to install dependencies
> that setup.py itself needs to run. Second, you also tell pip which build
> system you are using, once you have done that, setup.py is no longer
> necessary - instead, pip could install and then invoke flit for example.
>
> This isn't quite the situation you've outlined as setuptools will likely
> continue to have all its features. Instead, packages that don't use
> setuptools will also proliferate. It causes about the same problem for
> buildout without breaking older packages.
>
> In this scenario buildout will need to be able to install wheels however
> it wishes to do so because that is the output of the pluggable build
> system. I don't have advice for you on the implementation, except perhaps
> it could call out to pip.
>
> On Tue, Aug 16, 2016 at 12:59 PM Leonardo Rochael Almeida <
> leoroch...@gmail.com
> > wrote:
>
>> Spinning this off into its own thread...
>>
>> On 16 August 2016 at 13:10, Donald Stufft > > wrote:
>>
>>>
>>> On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida <
>>> leoroch...@gmail.com
>>> > wrote:
>>>
>>> Specifically, buildout right now has setuptools as its only dependence,
>>> and buildout uses it to locate, download and build sdist dependencies, or
>>> directly download and use .egg dependencies, which are important for
>>> Windows platforms in the communities that use buildout and depend on
>>> compiled extensions.
>>>
>>>
>>>
>>> Ah, I knew I was forgetting something. I think we should hold off on
>>> preventing egg uploads until setuptools can download and install wheels.
>>>
>>
>> Which brings us to a question that I'm meaning to ask for a while.
>>
>> It looks like we're close to removing all mentions of setuptools in pip.
>> When this happens, it looks like pressure is going to start to mount on
>> setuptools to drop the ability to install packages and limit itself on
>> being just a build tool.
>>
>> It makes sense that it should be so, considering that there would be no
>> incentive to keep around two distinct implementations of how to locate,
>> download and install stuff, one in pip (or whatever library pip uses for
>> locating and installing packages) and another in setuptools.
>>
>> In this future, I can imagine setuptools will simply complain loudly when
>> the packages it requires to do its work are not installed yet, perhaps with
>> a plugin mechanism for allowing another package to automatically resolve
>> and installing packages, (like today setuptools-git extends setuptools to
>> recognize files from git to be included in a package).
>>
>> And people will promptly write a pip implementation of this plugin (or
>> whatever it is that pip will use as a library to do it)... thereby
>> inverting the dependency between setuptools and pip.
>>
>> When this future arrives, buildout will be out of a mechanism for
>> locating, downloading and installing packages (not counting old versions of
>> setuptools, of course).
>>
>> Questions:
>>
>> 1. Is the future outlined above considered likely?
>>
>> 2. If this future arrives, what package should buildout be ported to use?
>> It should provide:
>>
>>  * The ability to locate packages and their versions, resolve their
>> dependencies, and download them, whether they're sdists or wheels (or eggs,
>> while we still have them)
>>  * Install each package separately into it's own directory that, if added
>> to `sys.path`, allows this same package (or perhaps another one) to read
>> information from entry points so that buildout can locate its own
>> extensions and install console scripts.
>>
>> Regards,
>>
>> Leo
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> 
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>

Re: [Distutils] Future of setuptools and buildout

2016-08-18 Thread Chris Barker 
On Wed, Aug 17, 2016 at 6:45 PM, Daniel Holth  wrote:

> And a while back I argued against setuptools-lite, because I thought it
> would not solve the poor extensibility problem that stems from its basic
> distutils derived design... which includes all the classes and subclasses
> that have to work together to do its thing. My own thinking is that
> setuptools is fine, except that when you need to extend it you have a big
> problem; so make it easy to replace.
>

I never had any fantasies that setuptool-lite would improve on this in a
way shape or form -- indeed, a better build system is likely to need to be
built from scratch, or on a totally different framework, anyway.


> Does pip monkey patch setuptools? I thought it just forced setuptools even
> if a package only asked for distutils.
>

good point -- not really a monkey patch, but it does force particular
configuration.


> So you are suggesting setuptools_lite would not honor the setup_requires =
> [] and test_requires = []
>

essentially, yes. Though It would be nice if that data could be available
for pip -- and I"m not sure how that is done at this point.


> and perhaps "setup.py install" would also throw an error?
>

Probably not -- but it could only install from local source -- like
distutils. Though if we could re-work it to only do develop mode and build
wheels, that may be good -- you'd need pip to install it.


> What else? I would suggest that to effectively argue the idea requires
> reading the source code and enumerating exactly which complexity goes away
> with the _lite version.
>

I don't know that reading the source is the issue -- I'm thinking entirely
in terms of stripping out the API, so there may be things that need to stay
in, but we don't expose them. though maybe even that's harder than I think.

But yes -- enumerating what would and wouldn't be supported would be a good
start. But here is the guideline:

It seems we have a (maybe vague) vision of what we want the tool ecosystem
to look like that involves separation of concerns, such that each component
is replaceable without having to change the others:

The concerns:
  - building a package
  - installing a package
  - dependency management
  - run-time version management
  - plugin handling, etc.
  - resource managment

setuptools does all of these, yes? but think of these in terms of when they
come into play:

build time:
   - building a package
install time: installing a package;
  - installing a package
  - dependency management
run time
  - run-time version management
  - plugin handling, etc.

so we have:

 - The mythical build tool
 - pip
 - maybe some of the setuptools spun off -- like pkg_resources

So why aren't we there yet?

One reason is that setuptools the only thing out there that does the
building and some of the run-time stuff. So people use it. And now you
can't replace any of it without making a mess.

However, pip_wheel have indeed addressed much of the install-time and
dependency management issues. So folks use them, and they work well.

So what problem do I want setuptool-lite to solve?

I actually have spent a lot of time using conda to do my package installing
and dependency management. So I want a way to build a python package that
JUST builds the package.

In practice, I've found (most) of the flags and work arounds to diable
etuptools "features", but really?

python setup.py install --single-version-externally-managed

Oh right, that's not enough, I need a record file:

setup.py install --single-version-externally-managed  --record record.txt

Then I want to install in develop mode:

python setup.py develop

OOPS! now it's easy installing a bunch of dependencies! arrgg!

OK, -- no-deps it is.

Google around, this kind of thing is a pain in a lot of people's butts
--even though it call all be worked around.

So my vision of setuptool-lite is that is simply does not do the things
that vision of a build tool should not do.

i.e.:

never easy-install anything!
don't try to resolve dependencies

others??

One "lite" version of setuptools that I personally think would be cool
> would be pkg_resources as a separate package on pypi. Setuptools would
> require it as a dependency.
>

Yes, I'd love to disable pkg_resources too, but couldn't until it was spun
off.

In fact, years ago, long before pip, setuptools drove me crazy with how it
combined run-time, build time and intall time stuff. A serious problem when
I wanted to bundle something up with py2app (or py2exe, or...).

So what would be accomplished?

package builders could simply put:

from setuptools_lite import setup

(etc.)

and then their user would never accidentally easy install anything, etc.
And it would serve as documentation / notification if a user was using a
what-should-be-deprecated feature.

By the way, this is totally acknowledged to be a hacky, temporary mess that
might only help a bit to get us closer from being locked into setuptools.

(though maybe it would be a

Re: [Distutils] Future of setuptools and buildout

2016-08-17 Thread Nick Coghlan 
On 18 August 2016 at 11:45, Daniel Holth  wrote:
> One "lite" version of setuptools that I personally think would be cool would
> be pkg_resources as a separate package on pypi. Setuptools would require it
> as a dependency.

I finally managed to relocate a discussion Jason, Donald and I had
about this a couple of months ago:
https://github.com/ncoghlan/pkg_resources_shim/issues/1

(The idea there was to make "pip install pkg_resources" just work by
implicitly installing setuptools - at the moment running that command
errors out, since Donald reserved the name on PyPI to prevent anyone
using it as an attack vector against folks actually looking for
setuptools)

Creating a standalone "pkg_resources" project turns out to be
relatively straightforward, but *removing* pkg_resources from
setuptools turns out to create significant setuptools bootstrapping
problems. That means the standalone project would need to use a
different name, and any associated bugs would subsequently need to be
fixed in two different projects, which seems like a bad idea.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

2016-08-17 Thread Nick Coghlan 
On 18 August 2016 at 08:18, Chris Barker  wrote:
> IF there were a setuptools_lite, user could simply do:
>
> import setuptools_lite as setuptools
>
> and they'd instantly get an error if they were using depreciated features,
> and their end users would never accidently easy install stuff :-)
>
> Anyway, this seem like a path forward, without having to wait for the future
> fabulous pluggable build system .

The problem with this is a pragmatic one rather than a philosophical
one, in that setuptools is just plain *hard* to work on, and doesn't
have a particularly robust test suite, so you can easily break other
people's usage while trying to fix the particular issue you personally
care about.

Hence the ongoing efforts to let people more easily use tools that
*aren't* setuptools, rather than attempting to disentangle and
modularise setuptools itself - the architectural challenges are such
that attempting to fix them would inevitably break existing usage,
while we can get most of the same benefits more transparently through
things like declarative dependencies for setup.py execution.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

2016-08-17 Thread Daniel Holth 
And a while back I argued against setuptools-lite, because I thought it
would not solve the poor extensibility problem that stems from its basic
distutils derived design... which includes all the classes and subclasses
that have to work together to do its thing. My own thinking is that
setuptools is fine, except that when you need to extend it you have a big
problem; so make it easy to replace.

For example, setuptools & distutils can't configure the compiler for
building extensions, without literally calling build_ext.run(). How
inconvenient.

Does pip monkey patch setuptools? I thought it just forced setuptools even
if a package only asked for distutils.

So you are suggesting setuptools_lite would not honor the setup_requires =
[] and test_requires = [] and perhaps "setup.py install" would also throw
an error? What else? I would suggest that to effectively argue the idea
requires reading the source code and enumerating exactly which complexity
goes away with the _lite version.

One "lite" version of setuptools that I personally think would be cool
would be pkg_resources as a separate package on pypi. Setuptools would
require it as a dependency.

My own setuptools-free C extension compiling demo continues to be pip
install -e hg+https://bitbucket.org/dholth/cryptacular#egg=cryptacular , in
a virtualenv.

On Wed, Aug 17, 2016 at 6:19 PM Chris Barker  wrote:

> Which brings us to a question that I'm meaning to ask for a while.
>>
>> It looks like we're close to removing all mentions of setuptools in pip.
>> When this happens, it looks like pressure is going to start to mount on
>> setuptools to drop the ability to install packages and limit itself on
>> being just a build tool.
>>
>>
> Yes, please!
>
> I argued  a while back for a setuptools-lite -- it would do all the things
> setuptools does that we think it _should_ do, and not do any of the others
> -- like it wouldn't easy-install anything EVER.
>
> most likely it would be a fork of the setuptools code with a bunch of
> stuff disabled. Or maybe even a setting in setuptools itself:
>
> import setuptools
> setuptools.disable_legacy
>
> from setuptools import setup, find_packages, ...
>
> Why
>
> Folks rely on the various "features" of setuptools, some important ones
> like buildout. So it's going to be a long time before we can deprecate all
> that in setuptools itself.
>
> But we're trying for a future with better separation of concerns --
> building, installing, packaging, run-time management.
>
> But as it stands now, people kind of HAVE to use setuptools to get desired
> build behaviour, but then their users an accidentally invoke features they
> don't want -- to teh point where pip goes in and monkey patches the darn
> thing.
>
> IF there were a setuptools_lite, user could simply do:
>
> import setuptools_lite as setuptools
>
> and they'd instantly get an error if they were using depreciated features,
> and their end users would never accidently easy install stuff :-)
>
> Anyway, this seem like a path forward, without having to wait for the
> future fabulous pluggable build system .
>
> -Chris
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

2016-08-17 Thread Chris Barker 
>
> Which brings us to a question that I'm meaning to ask for a while.
>
> It looks like we're close to removing all mentions of setuptools in pip.
> When this happens, it looks like pressure is going to start to mount on
> setuptools to drop the ability to install packages and limit itself on
> being just a build tool.
>
>
Yes, please!

I argued  a while back for a setuptools-lite -- it would do all the things
setuptools does that we think it _should_ do, and not do any of the others
-- like it wouldn't easy-install anything EVER.

most likely it would be a fork of the setuptools code with a bunch of stuff
disabled. Or maybe even a setting in setuptools itself:

import setuptools
setuptools.disable_legacy

from setuptools import setup, find_packages, ...

Why

Folks rely on the various "features" of setuptools, some important ones
like buildout. So it's going to be a long time before we can deprecate all
that in setuptools itself.

But we're trying for a future with better separation of concerns --
building, installing, packaging, run-time management.

But as it stands now, people kind of HAVE to use setuptools to get desired
build behaviour, but then their users an accidentally invoke features they
don't want -- to teh point where pip goes in and monkey patches the darn
thing.

IF there were a setuptools_lite, user could simply do:

import setuptools_lite as setuptools

and they'd instantly get an error if they were using depreciated features,
and their end users would never accidently easy install stuff :-)

Anyway, this seem like a path forward, without having to wait for the
future fabulous pluggable build system .

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

2016-08-16 Thread Paul Moore 
On 16 August 2016 at 17:58, Leonardo Rochael Almeida
 wrote:
> Questions:
>
> 1. Is the future outlined above considered likely?

I can't speak for setuptools, but certainly pip is intended to be the
canonical installer for Python. It's not impossible for competing
tools like the download/install features in setuptools to remain, but
like you say, I'd expect the trend to be against them.

> 2. If this future arrives, what package should buildout be ported to use?

That's really for buildout to decide.

> It should provide:
>
>  * The ability to locate packages and their versions, resolve their
> dependencies, and download them, whether they're sdists or wheels (or eggs,
> while we still have them)

pip will do this (no support for eggs exists or will be added, but
sdists and wheels yes). Note that pip should be run as a command line
tool - we don't support use of pip's internal API from an existing
Python process. If an in-process API is important, though, it wouldn't
be impossible to discuss adding that.

>  * Install each package separately into it's own directory that, if added to
> `sys.path`, allows this same package (or perhaps another one) to read
> information from entry points so that buildout can locate its own extensions
> and install console scripts.

That's not an installer feature. So pip doesn't provide anything
specific for that. Rather, that's a runtime support facility that is
tied to Python's import mechanisms. Setuptools (or rather,
pkg_resources?) may well continue to provide that, but laying out the
files present in a wheel in the format you need to support your
requirements would have to be added - maybe setuptools would be
interested in providing such a facility, I don't know. But it's not
exactly hard, at the most basic level you're probably only talking
about unzipping the wheel and writing a bit of metadata based on
what's specified in the wheel. Heck, a "wheel to egg" function
shouldn't be that hard to write, and then you could simply use the
existing runtime egg support code.

However, I don't believe that mandating that a single package should
provide both requirements is necessary, or helpful. The two features
are different - one of the features of setuptools that many people
(myself included) disliked, was the fact that it mixed "install time"
and "run time" features in one package, making it hard to avoid having
install time capabilities in a runtime-only environment.

Having said this, if someone were to propose, design and standardise a
*runtime* format for whatever you're hinting at as "install into its
own directory ... read information", and get it accepted as an
installation layout standard, then it might make sense to add support
to pip for installing files using that layout (analogous to the
"--target" flag for pip install). Runtime support would still have to
come from elsewhere, though. The design ideas for that spec may even
be based on the ideas developed for egg. But the key focus is moving
away from adhoc, implementation-defined standards, to properly
specified and documented standards. So step 1 would be to get such a
standard accepted, *then* look at proposing pip add it as an
alternative installation layout. (And I'd recommend not calling that
standard "egg" format - like it or not, the historical baggage
associated with that name will make it harder than you'd like to get
adoption of a new standard).

Hope this helps,
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

2016-08-16 Thread Donald Stufft 

> On Aug 16, 2016, at 12:58 PM, Leonardo Rochael Almeida  
> wrote:
> 
> Spinning this off into its own thread...
> 
> On 16 August 2016 at 13:10, Donald Stufft  > wrote:
> 
>> On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida > > wrote:
>> 
>> Specifically, buildout right now has setuptools as its only dependence, and 
>> buildout uses it to locate, download and build sdist dependencies, or 
>> directly download and use .egg dependencies, which are important for Windows 
>> platforms in the communities that use buildout and depend on compiled 
>> extensions.
> 
> 
> Ah, I knew I was forgetting something. I think we should hold off on 
> preventing egg uploads until setuptools can download and install wheels.
> 
> Which brings us to a question that I'm meaning to ask for a while.
> 
> It looks like we're close to removing all mentions of setuptools in pip. When 
> this happens, it looks like pressure is going to start to mount on setuptools 
> to drop the ability to install packages and limit itself on being just a 
> build tool.
> 
> It makes sense that it should be so, considering that there would be no 
> incentive to keep around two distinct implementations of how to locate, 
> download and install stuff, one in pip (or whatever library pip uses for 
> locating and installing packages) and another in setuptools.

So one of our end goals here (and why we’re so stuck on standards for 
everything, and not ad hoc things) is to make it entirely possible to have 
multiple distinct implementations of any part of the tool chain without having 
to try and copy each other’s features/implementation details to work. There’s 
one standard for what is correct and as long as you follow that, then you 
should work.

With that in mind, I certainly hope to get to a point where at the very least, 
setuptools make the conscious choice to continue to implement these features, 
rather than needing to do it because pip depends on it. Whether it makes sense 
for setuptools to continue to support them or not is something that the 
setuptools devs (particularly Jason) is going to be better suited to answer, 
but for my own opinion I hope we can get to a point that setuptools is just a 
build tool.

> 
> In this future, I can imagine setuptools will simply complain loudly when the 
> packages it requires to do its work are not installed yet, perhaps with a 
> plugin mechanism for allowing another package to automatically resolve and 
> installing packages, (like today setuptools-git extends setuptools to 
> recognize files from git to be included in a package).
> 
> And people will promptly write a pip implementation of this plugin (or 
> whatever it is that pip will use as a library to do it)... thereby inverting 
> the dependency between setuptools and pip.
> 
> When this future arrives, buildout will be out of a mechanism for locating, 
> downloading and installing packages (not counting old versions of setuptools, 
> of course).
> 
> Questions:
> 
> 1. Is the future outlined above considered likely?

I suspect that maybe, at some point it might be, but I think that setuptools 
will likely keep it’s support for these things for a good long while to avoid 
breaking things for people. At least until a satisfactory answer for what 
someone who is currently using setuptools should use emerges.

> 
> 2. If this future arrives, what package should buildout be ported to use? It 
> should provide:
> 
>  * The ability to locate packages and their versions, resolve their 
> dependencies, and download them, whether they're sdists or wheels (or eggs, 
> while we still have them)
>  * Install each package separately into it's own directory that, if added to 
> `sys.path`, allows this same package (or perhaps another one) to read 
> information from entry points so that buildout can locate its own extensions 
> and install console scripts.
> 


So you have a few choices, right now you can use pip in a subprocess (pip does 
not expose a programmatic API) or continue to use setuptools.

One of the things we’ve been trying to do with pip is instead of 
adding/exposing parts of pip via a programmatic API, is to instead extract them 
out into their own dedicated libraries that then pip consumes. This has the 
benefit that setuptools can depend on those things, instead of pip itself, but 
also that other people can come along and piece meal use the pieces that make 
sense for them. If someone has a great new idea for a pip-killer that does 
everything better, then they can do that, and reuse these libraries to help 
make their thing better right out of the starting gate.

So we could work on that effort too, I think the main thing holding it back is 
both developer time, as well as solid use cases so we can design the API around 
them. 

—
Donald Stufft



___
Distutils-SIG

Re: [Distutils] Future of setuptools and buildout

2016-08-16 Thread Daniel Holth 
There are two proposals to add pluggable build systems to Python sdists,
and one of them will probably be implemented. You add a pyproject.toml to
the root of your sdist, which the installer uses to install dependencies
that setup.py itself needs to run. Second, you also tell pip which build
system you are using, once you have done that, setup.py is no longer
necessary - instead, pip could install and then invoke flit for example.

This isn't quite the situation you've outlined as setuptools will likely
continue to have all its features. Instead, packages that don't use
setuptools will also proliferate. It causes about the same problem for
buildout without breaking older packages.

In this scenario buildout will need to be able to install wheels however it
wishes to do so because that is the output of the pluggable build system. I
don't have advice for you on the implementation, except perhaps it could
call out to pip.

On Tue, Aug 16, 2016 at 12:59 PM Leonardo Rochael Almeida <
leoroch...@gmail.com> wrote:

> Spinning this off into its own thread...
>
> On 16 August 2016 at 13:10, Donald Stufft  wrote:
>
>>
>> On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida <
>> leoroch...@gmail.com> wrote:
>>
>> Specifically, buildout right now has setuptools as its only dependence,
>> and buildout uses it to locate, download and build sdist dependencies, or
>> directly download and use .egg dependencies, which are important for
>> Windows platforms in the communities that use buildout and depend on
>> compiled extensions.
>>
>>
>>
>> Ah, I knew I was forgetting something. I think we should hold off on
>> preventing egg uploads until setuptools can download and install wheels.
>>
>
> Which brings us to a question that I'm meaning to ask for a while.
>
> It looks like we're close to removing all mentions of setuptools in pip.
> When this happens, it looks like pressure is going to start to mount on
> setuptools to drop the ability to install packages and limit itself on
> being just a build tool.
>
> It makes sense that it should be so, considering that there would be no
> incentive to keep around two distinct implementations of how to locate,
> download and install stuff, one in pip (or whatever library pip uses for
> locating and installing packages) and another in setuptools.
>
> In this future, I can imagine setuptools will simply complain loudly when
> the packages it requires to do its work are not installed yet, perhaps with
> a plugin mechanism for allowing another package to automatically resolve
> and installing packages, (like today setuptools-git extends setuptools to
> recognize files from git to be included in a package).
>
> And people will promptly write a pip implementation of this plugin (or
> whatever it is that pip will use as a library to do it)... thereby
> inverting the dependency between setuptools and pip.
>
> When this future arrives, buildout will be out of a mechanism for
> locating, downloading and installing packages (not counting old versions of
> setuptools, of course).
>
> Questions:
>
> 1. Is the future outlined above considered likely?
>
> 2. If this future arrives, what package should buildout be ported to use?
> It should provide:
>
>  * The ability to locate packages and their versions, resolve their
> dependencies, and download them, whether they're sdists or wheels (or eggs,
> while we still have them)
>  * Install each package separately into it's own directory that, if added
> to `sys.path`, allows this same package (or perhaps another one) to read
> information from entry points so that buildout can locate its own
> extensions and install console scripts.
>
> Regards,
>
> Leo
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig