Re: [Distutils] flit adapter for the current sdist format

[Thomas Kluyver]

Thanks Daniel, that's interesting to see.

For flit, I don't see a major benefit in generating sdists - for pure
Python packages, universal wheels are simpler and install quicker. I'd
really like to improve installing directly from a source tree, such as a
VCS checkout. To do that, any adapter would need to be checked into every
project using flit, so it's important to me that it be as small and simple
as possible. This was a key feature of Nathaniel's proposal for me: it
would let 'pip install .' work with just a few lines of straightforward
config.

Best wishes,
Thomas

On 29 October 2015 at 02:51, Daniel Holth  wrote:

> OK, so I wanted to see what it would take to create current-format sdists,
> defined as anything that has a setup.py that pip can install by running
> said setup.py. The following script takes a flit package (in the current
> directory and builds a pip-installable sdist. Tested against
> https://github.com/pohmelie/aioftp
>
> https://gist.github.com/dholth/e17f243085dcc539813f
>
> The first script is the archiver and the second script is the setup.py
> embedded into the sdist by the archiver. The embedded setup.py does an
> isolated install of flit into the setup-requires directory, if required,
> and uses flit to implement bdist_wheel. The trick is that the only thing
> pip really needs from egg_info is the current requirements, easily
> generated from wheel-style .dist-info.
>
> It might need slightly more metadata to pass on pypi and the argument
> handling is inflexible, but it demonstrates most of what would be required
> for a backwards compatible shim.
>
> If our new sdist format is an even easier way to adapt new build systems
> than writing this trivial setup.py shim, then we might have something.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Thomas Kluyver]

On 12 October 2015 at 05:37, Robert Collins 
wrote:

> And a bugfix :) - I didn't link to the docs for the build system
> interface we have today -
> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface
>

I'm happy to see that the command line interface pip requires from setup.py
is now documented. But the first thing it describes is the egg_info
command, and the description of what that involves is basically 'whatever
setuptools does'. Egg info is not any kind of standard, AFAIK - unlike dist
info. One of my main goals in writing flit is to build wheels without
involving setuptools at all, so I'm probably never going to implement that.

I also don't want to have a setup.py in the VCS at all, because it's an
invitation for people to run setup.py whatever, and then file bug reports
that it doesn't do exactly what setuptools/distutils would do.

This is what I like about Nathaniel's proposal. By looking for a new file,
it clearly indicates that this is a new style Python source tree, without
potential confusion over what setup.py does, and it lets us specify a
simple build system interface based on standardised pieces like wheels
rather than messing around with the details of setuptools.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Thomas Kluyver]

On 12 October 2015 at 11:01, Robert Collins 
wrote:

> Python packaging config in $root/pypa.yaml
>
> Defined keys:
> 
> ...
> build-tool:
>   # basic command that will spit a hunk of json back to the caller
> defining the
>   # commands to use with the build tool.
> 
>
> Build tool output json (in yaml format for ease of reading):
>

I would be quite happy with something along the lines of this proposal,
though I'd bikeshed about some of the details. I like the idea of the
source tree having a single reference to the build tool, and the build tool
describing itself to pip. I'd probably use references to Python
functions/modules rather than specifying shell-style commands, though; it
seems like there's less to go wrong that way.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Thomas Kluyver]

On 2 October 2015 at 12:58, Donald Stufft  wrote:

> The first thing that immediately stood out to me, is that it's recommending
> that downstream redistributors like Debian, Fedora, etc utilize Wheels
> instead
> of the sdist to build their packages from. However, that is not really
> going to
> fly with most (all?) of the downstream redistributors.
>

I can't now find that it in the draft text, but I don't think this is a
problem. This proposal means there's a standardised way to turn a source
tree into wheels. So downstream distributors can download an sdist - or
even a tarball of a VCS tag, if they're being strict about it - build
wheels from that using the config in this proposal, and then transform the
wheels into their own package format.

> Longer term, I think the answer is sdist 2.0 which has proper metadata
inside of it (name, version, dependencies, etc) but which also includes a
hook like this PEP has to specify the build system

I hadn't heard of this before - is it something that's being worked on?

Thanks,
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Add data-requires-python attribute to download links in simple repository API

[Thomas Kluyver]

In a discussion about how to allow pip to select a package version
compatible with the target Python version, Donald suggested adding a
data-requires-python attribute to links in the simple repository API,
which pip uses to find candidate downloads.

...

This would expose the Requires-Python metadata field already specified
in PEP 345. There is a separate PR for setuptools to allow specifying
this value.

I have opened a PR to add this to PEP 503, which defines the simple
repository API, and Donald asked that I post about it here for comments:

https://github.com/python/peps/pull/56

Thanks,
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Thomas Kluyver]

On Fri, Jul 15, 2016, at 01:25 AM, Donald Stufft wrote:
> Off the top of my head I can only really think of PIL, and *maybe*
> suds. Unless there’s a lot of these maybe all we really need is a
> policy for when administrators can/will edit the page to direct people
> towards a different project or a way to add an admin message directing
> people to another project.
 
Proposal: let's put some such manual intervention policy in place for now. 
Apply it for PIL to point to Pillow, and query the active suds forks to see if 
there's a generally agreed successor.
 
If this works well, great! If  the admins are flooded with 'successor
requests', then we can come back to question of an automated mechanism.
If there are too many abandoned packages with competing successors,
that's a trickier problem to solve, but at least we'd be considering it
with more information.
 
As further examples: pydot, pexpect and python-modernize have all been
unmaintained, leading to forks springing up. In all three cases, some of
the forkers eventually coordinated to contact the original maintainer,
get upload rights, and make new releases with the original name. It
would certainly have been nice if that could have happened sooner in
each case, but I doubt that any technical fix would have made a big
difference.
 
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] package made up of only .so's?

[Thomas Kluyver]

Hi Chris,

On 7 July 2016 at 11:49, Chris Withers  wrote:

> flit does look nice and clean, but appears to only support one
> module/package?
> Thomas, that ever likely to change?


That's by design, I'm afraid. I like one top-level module to correspond to
one installable distribution. Flit also assumes that the wheels it's
building are pure Python, so they get -none-any tags.

Feel free to cannibalise code from flit to build your specific wheels,
though. It's relatively easy to create a wheel without lots of tooling. For
instance, I have a script that unpacks the Windows binary installers for
PyQt4, reassembles the files into wheels, and uploads them to PyPI:
https://github.com/takluyver/pyqt4_windows_whl

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposal: "Install and save"

[Thomas Kluyver]

On Sat, Jul 23, 2016, at 02:32 PM, Alex Grönholm wrote:
> I'm -1 on this because requirements.txt is not really the standard way 
> to list dependencies.
> In the Python world, setup.py is the equivalent of Node's package.json. 
> But as it is
> Python code, it cannot so easily be programmatically modified.

Packaging based on declarative metadata:
http://flit.readthedocs.io/en/latest/


We have a bit of a divide. Specifying dependencies in setup.py (or
flit.ini, or upcoming pyproject.toml) is the standard for library and
tool packages that are intended to be published on PyPI and installed
with pip. requirements.txt is generally used for applications which will
be distributed or deployed by other means.

As I understand it, in the Javascript world package.json is used in both
cases. Is that something Python should try to emulate? Is it hard to
achieve given the limitations of setup.py that you pointed out?

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Thomas Kluyver]

On Mon, Aug 15, 2016, at 08:09 PM, Donald Stufft wrote:
> Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
> number of people still upload eggs, even though we have the wheel format.
> However, I think that we should (and generally do) consider eggs to be
> deprecated and while we don't want to break existing packages by removing
> them,
> we should block further uploads for them. Looking again at the download
> numbers
> eggs represented only 1.8% of total downloads in the last 30 days while
> wheels
> represented 41% and sdists represented 56%.

Is it possible to get the numbers for the proportion of uploads
represented by different formats?

I think this simplification is a good idea, but I expect there will be
some people frustrated when their egg uploads no longer work.

We still upload .zip sdists for some of our projects, in addition to
.tar.gz, but I think that's a habit left over from the days when it was
common to download and unpack sdists manually to run setup.py. I don't
think we lose anything if we drop the .zip downloads now.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Indexing modules in Python distributions

[Thomas Kluyver]

For a variety of reasons, I would like to build an index of what
modules/packages are contained in which distributions ('packages') on
PyPI. For instance:

- Identifying requirements by static analysis of code: 'import zmq' ->
requires pyzmq
- Finding corresponding packages from different packaging systems: pyzmq
on PyPI corresponds to pyzmq in conda, and python[3]-zmq in Debian
repositories. This is an oversimplification, but importable module names
provide a common basis to compare packages. I'd like a tool that could
pick between different ways of installing a given module.

People often assume that the import name is the same as the name on
PyPI. This is true in the vast majority of cases, but there's no
requirement that they are the same, and there are cases where they're
not - pyzmq is one example.

The metadata field 'Provides' is, according to PEP 314, intended for
this purpose, but the standard packaging tools don't make it easy to
use, and consequently very few packages specify it.

I have started putting together a tool to index wheels. It reads a .whl
file, finds modules inside it, and tries to identify namespace packages.
It's still quite rough, but it worked with the wheels I tried.
https://github.com/takluyver/wheeldex

Is this something that other people are interested in?

One thing I'm trying to work out at the moment is how the data would be
accessed: as a web service that tools can query online, or more like
Linux packaging, where tools download and cache a list to do lookups
locally. Or both? There's also, of course, the question of how the index
would be built and updated.

Thanks,
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Indexing modules in Python distributions

[Thomas Kluyver]

Thanks Steve, Chris,



On Tue, Feb 7, 2017, at 04:49 PM, Chris Wilcox wrote:

> I may be able to help jump-start this a bit and provide a platform for
> this to run on. I deployed a small service that scans PyPI to figure
> out statistics on Python 2 vs Python 3 support using PyPI Classifiers.
> The source is on GitHub:  https://github.com/crwilcox/PyPI-Gatherer.
> It watches the PyPI updates feed and refreshes entries for packages as
> they show up as modified. It should be possible to add your lib,
> query, and add an additional row or two to the result. I am happy to
> work together on this. Also, the data is stored in an Azure Table
> Storage which has rest endpoints (and a Python SDK) that makes getting
> the published data straight-forward.


I had a quick look through this, and it does look like it should provide
a useful framework for scanning PyPI and updating the results. :-)


What I'm proposing differs in that it would need to download files from
PyPI - basically all of them, if we're thorough about it. I imagine
that's going to involve a lot of data transfer. Do we know what order of
magnitude we're talking about? Is it so large that we should be thinking
of running the scanner in the same data centre as the file storage?


Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] recommended strategy for missing development tools

[Thomas Kluyver]

On Wed, Jan 25, 2017, at 10:23 AM, Robin Becker wrote:
> The setup.py in question complains if the
> extension files cannot be found, but does not error.

There was a change in pip at some point which means that it only shows
the output from setup.py if running it fails (i.e. it has a non-zero
exit status). I think this was largely because setup.py output is much
more verbose than you usually want, especially if pip installs many
packages at once.

It's nice in some cases where setup.py displays warnings that aren't
really a problem. In one project I maintain, a particular submodule
requires Python 3, while the rest of the code can be used on Python 2 as
well; people with older versions of pip have complained that it 'fails
to install', which it doesn't, because the warning looks like a fatal
error.

However, the flip side of this is that, as far as I know, setup.py has
to choose between succeeding silently, or failing and showing the user
an error message.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Virtalenv and setuptools [was: latest setuptools appears to require six in a breaking way]

[Thomas Kluyver]

On Wed, Jan 25, 2017, at 08:02 PM, Donald Stufft wrote:

> Since we have a PR up for PEP 518 already, 



Here's the link if anyone wants to take a look:

https://github.com/pypa/pip/pull/4144


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Virtalenv and setuptools [was: latest setuptools appears to require six in a breaking way]

[Thomas Kluyver]

On Wed, Jan 25, 2017, at 09:02 PM, Paul Moore wrote:
> Thanks. I've been seeing the PR comments in passing, but haven't
> really looked in detail. Is there anything I can do to help this
> along?

I believe it's working, but it's the first significant change I've made
in pip, so any review is welcome.

Thanks,
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] distlib and wheel metadata

[Thomas Kluyver]

On Thu, Feb 23, 2017, at 02:28 PM, Paul Moore wrote:
> I'd also drop "used to develop and deploy Python libraries,
> applications, and scripts" - why does what it's used for affect its
> category?

Things for working on & with Python code often have installation
requirements a bit different from other applications. E.g. pip installs
(or used to) with aliases specific to the Python version it runs on, so
pip, pip3 and pip-3.5 could all point to the same command. Clearly it
wouldn't make sense to do that for youtube-dl.

I'm not sure about 'tool' as a name for this category, but they often do
require different handling to general applications.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Indexing modules in Python distributions

[Thomas Kluyver]

On Wed, Feb 8, 2017, at 11:06 PM, Wes Turner wrote:

> So, IIUC,

> you're looking to emit

> ((URL, release, platform), namespaces_odict)

> for each new and all existing packages;

> by uncompressing every package and running every setup.py (hopefully
> in a container)?


Something like that, yes. For packages that publish wheels, we can
analyse those directly without needing to run setup.py. Of course there
are many packages with only sdists published.


> Could this (namespace extraction) be added to 'setup.py build' for
> the future?


Potentially. As I mentioned, there is a place in the metadata to put
this information - the 'Provides' field.  However, relying on package
uploaders would take a long time to build up decent coverage of the
available packages, so I'm inclined to focus on scanning PyPI, similar
to the tool Chris already showed.


Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Indexing modules in Python distributions

[Thomas Kluyver]

Thanks. So the current size is about 0.5 TB, and presumably if people
are maintaining full mirrors, PyPI itself can cope with that much
outgoing bandwidth being used.

Steve & Chris: does downloading & scanning that volume of data sound
like something you'd want to do on Azure? Does anyone there have some
time to put in to move this forwards?

Thomas

On Thu, Feb 9, 2017, at 10:18 PM, Jeremy Stanley wrote:
> On 2017-02-08 18:14:38 +0000 (+0000), Thomas Kluyver wrote:
> [...]
> > What I'm proposing differs in that it would need to download files from
> > PyPI - basically all of them, if we're thorough about it. I imagine
> > that's going to involve a lot of data transfer. Do we know what order of
> > magnitude we're talking about?
> [...]
> 
> The crowd I run with uses https://pypi.org/project/bandersnatch/ to
> maintain a full PyPI mirror for our project's distributed CI system,
> and du says the current aggregate size is 488GiB. Also if you want
> to initialize a full mirror this way, plan for it to take several
> days to populate.
> -- 
> Jeremy Stanley
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] distlib and wheel metadata

[Thomas Kluyver]

On Wed, Feb 15, 2017, at 03:40 PM, Daniel Holth wrote:

> It would make sense to go ahead and delete the obsolete fields, I'm
> sure they were overlooked because they are not common in the wild.
> 

> From PEP 345:

>  * Deprecated fields:
>* Requires (in favor of Requires-Dist)
>* Provides (in favor of Provides-Dist)
For reference, packages made with flit do use 'Provides' to indicate the
name of the importable module or package that the distribution installs.
This seems to me to be something worth exposing - in another thread,
we're discussing downloading and scanning packages to get this
information. But I accept that it's not very useful while only a tiny
minority of packages do it.


Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Thomas Kluyver]

On Sun, Aug 21, 2016, at 05:19 PM, Jim Fulton wrote:
> I'm open to shelling out, but pessimistic that it will turn out well.
> I started with that approach initially with easy_install and it fell
> apart quickly.  But when we get into it... who knows?

I think shelling out is a reasonable approach for certain operations
if the tool you're shelling out to is designed for it. When we write
command-line programs, we often think about users typing in commands
at a terminal, and design things that are flexible and somewhat
smart. If shelling out is to be an API, the commands need to be
predictable and dumb.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Thomas Kluyver]

On Mon, Aug 22, 2016, at 07:15 AM, Sylvain Corlay wrote:
> I find this worrying that the main arguments to not include a patch
> would be that
>
>  - this part of the standard library is not very maintained (things
>don't get merged)
>  - earlier versions of won't have it

Would it make sense to add it to both distutils and setuptools? The
standard library continues to evolve, projects that require Python 3.6
wouldn't need to use setuptools, but we could start using it sooner.
There's obviously some cost in code duplication; I haven't looked at the
code in question, so I don't know how bad this is.

I've run into this argument before when trying to change things in 
non-packaging-
related parts of the stdlib, and I agree with Sylvain that it's
fundamentally problematic. If we're trying to improve the stdlib, we're
obviously taking a long view, but that's how we ensure the stdlib is
still useful in a few years time. This goes for packaging tools as much
as anything else.

I already have projects where I'm happy to require Python >=3.4, so
being able to depend on Python 3.6 is not such a distant prospect.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 527 - Removing Un(der)used file types/extensions on PyPI

[Thomas Kluyver]

On Thu, Aug 25, 2016, at 12:30 PM, Nick Coghlan wrote:
> My one remaining question would be whether or not we have any projects
> that are in the habit of uploading both .zip and .tar.gz for the same
> release - if there are, the restriction to only one sdist per release

IPython's release script uploads both sdist formats:
https://pypi.python.org/pypi/ipython

Several of our other IPython/Jupyter packages have both formats on PyPI.
I don't know how unusual we are.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires: the obvious option(?)

[Thomas Kluyver]

On Tue, Aug 30, 2016, at 05:51 PM, Antony Lee wrote:
> I am not really a fan of PEP518 in general.  Basically, the idea of
> setup.py is that declarative languages are not sufficient to express a
> build system (and AFAICT this is always going to be the case for
> expressing, say, compiler flags for extensions), so I'd rather just
> accept that and stick everything in setup.py instead of adding more
> parameter files.  What if someone wants dynamic build dependencies?

Dynamic build deps aren't precluded - the idea is that the build system
can discover additional dependencies when it runs, while the static build-
system field specifies just what's required to run the build system
itself.  However, the build system interface was split out into separate
PEPs (517 & 516 are alternatives) to allow 518 to go forwards.

I take totally the opposite view: we should make as much metadata as
possible declarative, even though we know we can't define a totally
general build system with purely declarative information.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Best way to handle packaging a project that only applies to certain versions of Python

[Thomas Kluyver]

There is also the 'python requires' metadata, which can now be provided
via setuptools, but I think pip 8.2 needs to be released before that is
respected.


On Tue, Aug 30, 2016, at 01:08 AM, Daniel Holth wrote:
> Let's say you have different code for python 3.3 and python 3.4+. Tag
> one wheel py33-none-any and the second py34-none-any. The second wheel
> is preferred on python 3.4 and above, but ignored by 3.3. The py3 tag
> wouldn't work well here.
> Order of preference on 3.5 is: ('py35', 'none', 'any'), ('py3',
> 'none', 'any'), ('py34', 'none', 'any'), ('py33', 'none', 'any'),
> ('py32', 'none', 'any'), ('py31', 'none', 'any'), ('py30',
> 'none', 'any')
> Enscons would let you control the wheel tag directly but doesn't yet
> let you build multiple wheels with a single invocation.
>
> On Mon, Aug 29, 2016, 19:46 Donald Stufft  wrote:
>>
>>> On Aug 29, 2016, at 7:34 PM, Brett Cannon  wrote:
>>>
>>> Someone has asked that I do a new release of importlib that includes
>>> a LICENSE file on PyPI: https://pypi.org/project/importlib/.
>>> Historically I have had the setup.py simply not include any Python
>>> code when built on versions of Python that include importlib in the
>>> stdlib itself:
>>> https://github.com/brettcannon/importlib/blob/master/setup.py .
>>>
>>> But now I would like to do a wheel. Is there some way I'm not
>>> thinking of to have a wheel that will leave out code or not install
>>> itself if a certain version of Python is used? Or will the user have
>>> to specify a proper Python requirement in their requirements.txt to
>>> get that kind of experience?
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>> If your setup.py produces different output on different versions of
>> Python, you’ll need multiple wheels.
>>
>> —
>>
>> Donald Stufft
>> ___
>>  Distutils-SIG maillist  - Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
> _
> Distutils-SIG maillist  - Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Module Installation Issues

[Thomas Kluyver]

On Wed, Sep 14, 2016, at 04:31 PM, Paul Moore wrote:
> Of course, it doesn't solve the problem of a user who doesn't know
> what a "shell prompt" is...

Especially since the interactive Python interpreter is also a shell
prompt. The title of an IDLE window on my system is "Python 3.5.2 Shell"
;-). I'd suggest the terminology "system command prompt" to refer to...
that thing that you should type 'pip' into.

Even if users don't know exactly what it means, indicating that they're
running something in the wrong place gives them a better chance of
working out what to do.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Module Installation Issues

[Thomas Kluyver]

On Tue, Sep 13, 2016, at 08:39 PM, Steve Dower wrote:
> It would help if you could post the full error output (sanitizing
> paths if needed). But you may just need to upgrade pip (python -m
> install -U pip).

I think Ryan may have typed that command at a Python prompt rather than
a system command prompt. Unfortunately the distinction often isn't
clear in examples, because the experienced developers writing the
instructions are used to guessing which commands are Python and which
are system commands.

One thing I'd quite like to see Python grow is a standard function to
install packages from inside Python. In R, the install.packages()
function is the default way to get third party packages, and I think
staying in one interactive prompt does make things easier for new users
to understand.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Module Installation Issues

[Thomas Kluyver]

On Thu, Sep 15, 2016, at 01:53 PM, Wes Turner wrote:
> Shell commands can/could/should be clearly indicated with a '$'
> prefix:
>
> $ python -m pip install numpy

That's a common convention, but:

- It comes from POSIX platforms where the default prompt ends in $. The
  default prompt in Windows ends with >, so $ is not clear.
- On all platforms, new users may well not recognise that convention.
- Users may think they have to type the $ as part of the command,
  leading to even more confusion.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Thomas Kluyver]

On Mon, Aug 15, 2016, at 08:40 PM, Donald Stufft wrote:
> Or were you wondering for the last 30 days like I did for downloads? If
> so then:
> 
> * sdist: 15601 (66%)
> * bdist_wheel: 6398 (27%)
> * bdist_egg: 1076 (4.5%)
> * bdist_dumb: 195 (0.8%)
> * bdist_wininst: 167 (0.7%)
> * bdist_rpm: 38 (0.1%)
> * bdist_msi: 9 (0.03%)
> * Everything Else: 0 (0%)

Sorry for the lack of clarity, I was indeed looking for these numbers
for recent uploads.

This suggests there is a small but non-negligible group still uploading
eggs. Is it worth looking at what projects are doing this, and
potentially contacting some authors to understand why?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 527 - Removing Un(der)used file types/extensions on PyPI

[Thomas Kluyver]

On Thu, Aug 25, 2016, at 05:29 PM, Nick Coghlan wrote:
> Could you give a bit more detail on how you came to be publishing
> both? The main thing we're trying to avoid is missing a practical use
> case for the status quo where folks can upload both - if it's just an
> artifact of Windows and *nix having different default formats, then
> the convergence in distutils and setuptools will fix it implicitly,
> but if it's a deliberate design decision, then we need to check if
> that's based on a misunderstanding of how pip/easy_install/et al
> consume the two formats.

I think that script was created before my time in the project. I'd guess
it's just a historical artefact, but Fernando might know more.

Fernando: is there a reason we publish both .zip and .tar.gz sdists for
each release? PyPA is thinking of only allowing one sdist per release.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

On Wed, Nov 23, 2016, at 07:22 PM, Donald Stufft wrote:
> I think at a minimum we should get PEP 518 support into pip first. I
> don't think layering more things on top of a PEP that isn't yet
> implemented is a good approach to this. 

I went to make a start on this, but I got stuck on whether and how pip
should create temporary environments to install build dependencies and
build wheels. I've posted more details on the issue:

https://github.com/pypa/pip/issues/3691
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP 517: Build system API

[Thomas Kluyver]

I'd like to push PEP 517 forwards again. This PEP specifies a general
build system interface so that a source tree can specify a tool (such as
flit), and pip can ask that tool to generate a wheel. This would allow
us to install from an sdist or a VCS checkout without running a setup.py
file.

https://www.python.org/dev/peps/pep-0517/

Developments since the last time this was discussed (to my knowledge):
- It now uses the pyproject.toml file specified in PEP 518, which was
already accepted. 517 originally specified a JSON file; 518 explains why
TOML is preferred (basically: comments).
- I have implemented the proposed build-system API in a pull request for
Flit; this has been fairly straightforward so far.
https://github.com/takluyver/flit/pull/77

Questions:
1. Editable installs. The PEP currenly specifies a hook to do an
editable install (like 'pip install -e' or 'setup.py develop') into a
given prefix. I don't think that specifying a prefix is sufficient to
cover '--user' installation, which uses a different install scheme,
especially on Windows and OSX framework builds. We can:
a. Add an extra parameter 'user' to the hook, to override the prefix and
do a user install.
b. Leave it as is, and do not support editable user installation (which
would make me sad, as I do editable user installs regularly)
c. Decide that editable installs are too fiddly to standardise, and
leave it to users to invoke a tool directly to do an editable install.

2. Dash vs. underscore, bikeshed reloaded! Currently,  the table name
uses a dash: [build-system], but the key added by PEP 517 uses an
underscore: build_backend. This seems a bit messy. I propose that we
change build_backend to build-backend for consistency. Dashes and
underscores can both be used in a TOML key without needing quoting.

Thanks,
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

On Thu, Nov 24, 2016, at 08:07 PM, Paul Moore wrote:
> Just curious - how does flit handle Windows for this? Symlinks aren't
> really an option there (you need elevation to create a symlink).
> Paul

It largely doesn't at present; it started out as a personal tool for me,
and I mostly use Linux. I have a few ideas about how to deal with
Windows, though:

- There's a feature called NTFS Junction Points, which is supposed to be
like symlinks, but only for directories. I believe these can be created
by regular users without elevated permissions. It might be possible to
use this for packages, albeit not for single file directories.
- Create a script/tool to give a user the permission bit that allows
creating symlinks, so users who know what they're doing can enable it.
Maybe there's some reason it's admin-only by default, though? It's safe
enough for normal users on Unix, but I don't know how different Windows
is.
- Run a daemon to watch package folders and reinstall on any changes,
something like Nathaniel mentioned with 'pip watch'. I probably wouldn't
do this as part of flit, but I'd be happy to see it as a separate tool.
As Nathaniel pointed out, this can actually support more stuff than
setuptools editable installs, or my symlinks, because it can re-run
build steps.

Thomas

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

On Thu, Nov 24, 2016, at 07:50 PM, Thomas Kluyver wrote:

> I made a PR to flit to handle this case better in uninstallation.

I meant a PR to *pip*.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

I've made PRs against PEP 517 for:

Underscore to dash in build-backend:
https://github.com/python/peps/pull/139

1a: Add a user parameter to the install_editable hook
https://github.com/python/peps/pull/140

OR:

1c: Get rid of the install_editable hook
https://github.com/python/peps/pull/141

I'm leaning towards 1c; it can always be standardised in a separate PEP,
and an optional hook added back.

Thomas

On Thu, Nov 24, 2016, at 09:06 AM, Paul Moore wrote:
> On 24 November 2016 at 00:32, Nathaniel Smith  wrote:
> > Also note that just like we decided to split the basic pyproject.toml
> > proposal (now PEP 518) from the build system interface proposal (now
> > PEP 517), it might (probably) makes sense to split the editable
> > install part of the build system interface proposal from the rest,
> > just in the interests of making incremental progress.
> 
> This would be my preference. I'm assuming (again, incremental
> progress) that "pip install -e" will still work as at present for
> projects that don't use the new infrastructure, so it's hardly an
> urgent need.
> 
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Interrogate distribution for an entry point command path

[Thomas Kluyver]

On Sat, Oct 15, 2016, at 06:57 PM, Ben Finney wrote:
> I'm modifying an existing application that invokes the program as a
> subprocess, so I'm wanting to find that program as an external command.

If the entry point looks like:

foo=foomod:main

Then you can invoke it in a subprocess by running:

subprocess.Popen([sys.executable, '-c', 'import foomod; foomod.main()'])

This avoids the need to work out where the 'foo' script has been
installed to.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

On Thu, Nov 24, 2016, at 11:10 PM, Paul Moore wrote:
> Honestly, I don't see what's so bad about pth files. They are a
> standard supported Python approach. Maybe setuptools' use of them is
> messy? I recall it was possible to end up with a lot of clutter, but
> that was going back to the egg days, I'm not sure if it's like that
> any more. Why not just have a single pth file, maintained by the build
> tool, for all editable installs? Most users would then only have one
> or two pth files.

They would definitely be a lot more tolerable if it wasn't for
setuptools' use of the execution loophole, and indeed that misfeature as
a whole.

But even without that, they're not ideal. I routinely have development
installs of quite a lot of different packages. If each one of those is
an entry on sys.path from a .pth file, there's a performance penalty, as
every 'import qux' has to stat() in dozens of directories which don't
contain qux at all:

stat('.../foo/qux.py')
stat('.../foo/qux/__init__.py')
stat('.../foo/qux.so')
stat('.../bar/qux.py')
stat('.../bar/qux/__init__.py')
...
stat('.../qux/qux.py')   # At last!

If you put tooling scripts/modules in the root of your project
directory, it also increases the chance of name clashes.

Maybe we should look into something more like symlinks for doing
editable installs, so a file called '.../site-packages/qux.pylink' would
contain the path to the qux module/package. This would only be read on
'import qux', so it wouldn't affect other imports at all.

Brett has merged my PR for option 1c, so PEP 517 no longer specifies a
hook for editable installs. If we need extra hooks to support editable
installs, we can specify them separately in a later PEP.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] obtaining project name and packages

[Thomas Kluyver]

I have a tool that does this from a wheel:
https://github.com/takluyver/wheeldex

>From an sdist, I think you need to either build a wheel or install it
before you can get this information reliably.

Some of my installed packages have a 'top_level.txt' file in the
.dist-info folder, containing a list of the top-level package names
installed by that distribution. I don't believe this is formally
specified anywhere, though, and packages created by flit do not have it.

Thomas

On Wed, Mar 29, 2017, at 07:41 PM, Chris Jerdonek wrote:
> Hi, this seems like a simple question, but I haven't been able to find
> the answer online:
> 
> What is the current recommended way to get (1) the name of a project,
> and (2) the names of the top-level packages installed by a project
> (not counting the project's dependencies).  You have access to / can
> run the project's setup.py, and you're also allowed to assume that the
> project is installed.
> 
> For example, for (1) I know you can do--
> 
> $ python setup.py --name
> 
> But I'm not sure if accessing setup.py is no longer recommended (as
> opposed to going through a tool like pip).
> 
> Thanks a lot,
> --Chris
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] reproducible builds

[Thomas Kluyver]

Flit already supports $SOURCE_DATE_EPOCH for building wheels.

I think the environment variable is a good idea: if it gets wide
support, you will be able to set a single thing to affect lots of
different build tools, rather than working out where you need to add
command line arguments to half a dozen different build steps.

Thomas

On Fri, Mar 17, 2017, at 05:33 PM, Matthias Bussonnier wrote:
> On Fri, Mar 17, 2017 at 10:19 AM, Robin Becker 
> wrote:
> > An issue has been raised for reportlab to support a specific environment
> > variable namely SOURCE_DATE_EPOCH. The intent is that we should get our time
> > from this variable rather than time.localtime(time.time()) so that produced
> > documents are more invariant.
> >
> > First off is this a reasonable request? The variable is defined by debian
> > here https://reproducible-builds.org/specs/source-date-epoch/
> >
> > What happens if other distros decide not to use this environment variable?
> > Do I really want distro specific code in the package?
> 
> For what it is worth, it seem like it will make its way into CPython as
> well:
> https://github.com/python/cpython/pull/296
> 
> And IFAICT, this env variable naming is already more than just debian.
> 
> -- 
> M
> 
> 
> >
> > In addition we already have our own mechanism for making the produced
> > documents
> > invariant although it might require an extension to support externally
> > specified date & time as in the debian variable.
> >
> > In short where does the distro responsibility and package maintainers
> > boundary need to be?
> > --
> > Robin Becker
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] reproducible builds

[Thomas Kluyver]

On Mon, Mar 20, 2017, at 01:02 PM, Robin Becker wrote:
> Well now I am confused. The date / times mentioned in the debian patch
> are those 
> we force into the documents produced by the reportlab package when it is
> used.
> 
> They would not normally be part of the package itself. Although the
> reportlab 
> documentation is available in the source I'm fairly sure we don't include
> it in 
> the wheels.

I'm guessing, but I imagine that Debian may be using reportlab in the
builds of other packages, to build documentation. It's normal for Debian
packages to include built docs, unlike wheels. So they would want it to
create PDFs reproducibly, but the PDFs generated in your test suite
probably don't matter.

> I guess the algorithm variation across pythons would make dictionary order 
> quite variable.

For a Python based tool, I think it's reasonable that reproducing a
build requires running with the same version of Python.

The requirement would be that, with enough information about the build
environment, you *can* produce an identical PDF. It needn't (AFAIK) be
identical every time anyone builds it.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Idea: Using Trove classifiers for platform compatibility warnings

[Thomas Kluyver]

On Sat, Apr 8, 2017, at 03:17 AM, Nick Coghlan wrote:
> 2. At least one "Programming Language :: Python" tag is set, but the
> tags don't include any that cover the *current* Python version

We've just enabled setuptools, PyPI and pip to use the python-requires
metadata field, so that (for instance), IPython 6.0 was uploaded with
python-requires: >=3.3 . Pip 9 respects that, so users running 'pip
install ipython' on Python 2.7 will get the latest IPython 5.x release.

So we have a relatively elegant way to tell pip about supported Python
versions - although it's designed for the use case "this won't work on
X.Y, so don't even try to install it", rather than "this isn't
explicitly marked as supporting X.Y, but it might work anyway."

Personally I try to avoid the 'Python :: X.Y' classifiers because I
don't want to be updating a dozen packages every time a new Python comes
out. I've used the 'Python :: 2'/'Python :: 3' classifiers before - but
even those could cause a lot of inaccurate metadata if Python 4 happens
and is mostly compatible with Python 3.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Thomas Kluyver]

On Sat, Jul 29, 2017, at 09:47 AM, Paul Moore wrote:
> +1 from me on just adding the project root to sys.path.

Ditto. I don't expect most projects to use it, but it seems like a handy
bit of flexibility to have available. The potential clutter is a bit
ugly, but I don't think it's worth the trouble of specifying an extra
key to avoid that.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Thomas Kluyver]

On Tue, Aug 15, 2017, at 04:42 PM, Brett Cannon wrote:
> I'm also willing to bet that Thomas is busy prepping for JupyterCon.

Sadly I won't be at JupyterCon. I'm avoiding all travel to the US in the
current political situation.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Tue, Jul 11, 2017, at 04:48 PM, Daniel Holth wrote:
> Pro_duced.
> 
> Mandatory backend hooks:
> 
> - build_sdist(sdist_directory, config_settings={})
> - build_wheel(wheel_directory, config_settings={}, build_directory=None, 
> metadata_directory=None)> 
> Optional backend hooks:
> 
> - get_requires_for_build_sdist(config_settings={})
> - get_requires_for_build_wheel(config_settings={})
> - prepare_metadata_for_build_wheel(metadata_directory, config_settings={})> 

Thanks Daniel! I'm happy with all of those names.

I'm at Europython at the moment, so I may not have much time to work on it for 
the next few days, but if no-one else gets there first I'll have a go at 
updating the PEP with the latest changes.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Fri, Jul 14, 2017, at 11:59 AM, Nathaniel Smith wrote:
> What are the advantages of this design, as compared to the
> fallback of doing unconditional copytree (like pip does now and may
> well continue doing for years to come regardless of what we say here),

I don't think pip currently does copytree. Naively copying everything
may be very slow, especially on Windows, because it includes things like
the .git subdirectory. If you exclude the .git subdirectory, you break
tools like setuptools_scm which rely on the VCS for version numbers.

> or the slightly fancier fallback that my draft supports of attempting
> to build an sdist and if that fails doing a copytree instead?

Having a fallback which may be pathologically slow seems like bad UX
(it's hard to see what's going wrong), and I don't want whatever is
specified to hinge on generating sdists.

> ...and if pip's goal is to go via sdist whenever possible while always
> being careful never to modify the source tree, then why did we end up
> with a design where sdist generation is the one case that *is*
> encouraged to modify the source tree? This doesn't make any sense.

Sdist generation is *not* encouraged to modify the source tree. It is
encouraged to avoid generating significant artifacts, because the build
steps should be done after extracting the sdist. But if the backend
wants to generate intermediates during sdist creation, it is free to
make a temporary directory, or use an external cache mechanism.

> I'm sorry for dragging this out -- I know you just want to get
> something finished and stop arguing about this. So do I :-). And your
> efforts to keep pushing this stone up the hill are much appreciated. I
> just think we should... find a shorter hill and declare victory over
> that.

I appreciate the enthusiasm, but I feel like we're arguing about whether
that hill over there is smaller when we're almost at the top of this
one. ;-)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Mon, Jul 17, 2017, at 01:07 PM, Paul Moore wrote:
> If we have a consensus here that "build a sdist and build a wheel from
> it" is an acceptable/viable main route for pip to generate wheels
> (with "just ask the backend" as fallback) then I'm OK with not
> bothering with an "ask the backend to build a wheel out of tree"
> option. My recollection of the history was that there was some
> resistance in the past to pip going down the "build via sdist" route,
> but if that's now considered OK in this forum, then I'm fine with
> assuming that either I was mistaken or things have changed.

I think I was one of the people arguing against going via an sdist. The
important point for me is that an sdist is not a requirement for
installing from source -  it's ok by me if it tries building an sdist
first and then falls back to building a wheel directly.

Since flit generates no intermediate artifacts, though, it's not an
informative data point for where they should be placed.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Mon, Jul 17, 2017, at 02:56 PM, Nick Coghlan wrote:
> If Thomas is OK with it, I'd be fine with using "raise
> NotImplementedError" for that purpose.

I will implement this if it's what we decide, but I agree with Nathaniel
that a sentinel value is probably more robust, because return values
can't automatically bubble up from internal code like errors do.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

To summarise the current state of this discussion as I understand it,
the hooks we are currently thinking of defining are:

def get_build_wheel_requires(config_settings)
def prepare_wheel_metadata(metadata_directory, config_settings)
def build_wheel(wheel_directory, config_settings, build_directory=None,
metadata_directory=None)

def get_build_sdist_requires(config_settings)
def build_sdist(sdist_directory, config_settings[,
build_directory=None])  # Last param under discussion

I know we were also discussing a different naming scheme, but I've
forgotten what names were proposed, and don't have time to find the
relevant email again. Could someone produce a copy of the list above
with the proposed names substituted?

Thanks,
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Mon, Jul 10, 2017, at 07:01 AM, Nick Coghlan wrote:
> So I think we have pretty solid evidence that the reason the
> procedural "build directory preparation" hook wasn't sitting well with
> people was because that isn't the way build systems typically model
> the concept, while a "build directory" setting is very common (even if
> that "setting" is "the current working directory when configuring or
> running the build").

Hooray! :-)

Do we want to also provide a build_directory for the build_sdist hook?
In principle, I don't think making an sdist should involve a build step,
but I know that some projects do perform steps like cython code gen or
JS minification before making the sdist. I think this was a workaround
to ease installation before wheel support was widespread, and I'd be
inclined to discourage it now, so my preference would be no
build_directory parameter for build_sdist. Backends which insist on
generating intermediates at that point can make a temp dir themselves.

Then I guess that the choice between building a wheel directly and
attempting to build an sdist first (with direct fallback) is one for
frontends, and doesn't need to be specified.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Mon, Jul 10, 2017, at 04:13 PM, Nick Coghlan wrote:
> My apologies folks, this is an entirely irrelevant tangent brought on
> by my attempting to explain my own preference that source archives
> (including sdists) actually *be* source archives, containing solely
> the original software is *its preferred form for modification*.

I also agree with this view - ideally, I think there should be no
generated files in an sdist besides the metadata needed for tools
consuming it.

Many packages have included generated files to simplify building from an
sdist, but I see this as a symptom of poor infrastructure which we're
gradually fixing. In particular, before we had wheels, installing a
package nearly always meant building it, so there was a strong pressure
to do part of the build process before distributing the source.

I don't think it's practical to forbid generating files to put in the
sdist, but for pep517 I'd say it's appropriate to gently discourage it
by not providing a build directory to that hook. I'm not going to argue
hard for this if other people think the sdist hook needs that parameter,
though.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Thu, Jul 6, 2017, at 07:19 PM, Paul Moore wrote:
> That's a good point - and provides a good contrast to my perspective
> as a pip developer that *pip* gets issues raised that aren't really
> pip's problem. I think it's in everyone's best interests to ensure
> that the user's experience is as unambiguous as possible in saying
> where any given problem lies.

Working on Jupyter, I've also seen plenty of misattributed issues from
people who think that we make the whole ecosystem they're using in the
notebook. I'm all for making it as unambiguous as possible, but I also
believe that in many cases it's impossible to be totally clear which
part has gone wrong, especially for users unfamiliar with the stack. So
my priority is to minimise user-facing failures, because we're all
likely to get bug reports.

> One thought occurs to me in that context - in my view, we should be
> clearly presenting to the user that it's *pip's* role to do the
> install, and flit's responsibility is to build wheels. I know that
> flit includes an install command, but I view that as a temporary
> workaround for the fact that PEP 517 isn't implemented yet. I'd be
> interested to know if you agree with that.

Yes-ish. There are three parts:

- flit installfrom (e.g. to install from a Github repo): entirely a
stopgap measure, I'm very happy to pass this responsibility over to pip.
- flit install (local install): I'll probably recommend pip once that
works, but may leave it in place a bit longer. It already works by
building a wheel and asking pip to install it.
- flit install --symlink (development install): stays around at least
until there's a standardised approach for this that I'm happy with.

> But essentially, we're promoting "pip install
> " as the canonical install command, and "pip wheel
> " as the canonical "build a wheel" command - backend
> specific commands should be for specialised use only, as I see it.

Depending on exactly what you mean by 'specialised'. I don't see flit as
'a PEP 517 backend', but rather 'a tool which provides a PEP 517
backend'. I will continue to recommend that developers invoke flit
directly to build and publish packages, but it should be transparent to
typical downstream users.

> Personally I wouldn't have a major problem with this, although I don't
> think Donald would agree, as there's questions that he's raised around
> potential inconsistencies between sdists and wheels built direct from
> the source tree that are unanswered in this model. My biggest concern,
> though, is that if we take this view, then it's critical that we have
> a reliable and efficient means of *copying* source trees.

So we have two alternative proposals for this bit:

1. Try to make an sdist, fall back to copytree if not supported
+ Provides some measure of built-in checking for the sdist problem
+ Reuses existing sdist machinery
- Fallback may be slow

2. Separate hook for efficient copy
+ Single mechanism is more predictable than primary+fallback
+ Reliably efficient
- Requires extra backend code

I'm willing to implement the necessary for either (but preferably not
both!). I think 2 is perhaps a bit more user friendly - it's not going
to be inexplicably slow because you've hit the fallback case.

> Tox may have more stringent requirements - currently it requires the
> ability to build a sdist to install from, and I'm inclined to think
> that this is a deliberate design choice rather than merely a
> convenience. I'm guessing that no-one has particularly explored the
> question of how tox would interact with flit-based projects yet?

I don't think so.

> Would
> it be acceptable to say that tox only works on a full checkout with
> VCS tools present (i.e, what flit needs to build a sdist) for
> flit-based projects? I don't really know.

I'd be willing to explore whether we can do better than that, but I see
tox as a developer tool, so I wouldn't consider it a show-stopper if it
required the presence of a VCS.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Thu, Jul 6, 2017, at 06:19 PM, Donald Stufft wrote:
> I *think* if we had some way to signal expected failure vs unexpected failure 
> this would be reasonable to me. I wouldn’t just want it to flat out be any 
> failure, but if we used Nathaniels NotImplemented idea or something similar 
> to indicate that “hey, I can’t build an sdist here for expected reasons” 
> compared to “Hey I tried to build the sdist, but something went wrong” I 
> think that would be workable.
I'd prefer that it catches any failure, prints a warning and carries on to the 
fallback, but I can see where you're coming from, and I can live with this if I 
can trigger the fallback when the VCS is not available.
> A similar-ish scenario is I hope to in the future be able to start validating 
> the rendering of long_description on PyPI on upload, and rejecting for 
> invalid syntax, because while readme_renderer exists and people can use it 
> (and it lets them detect problems earlier on) forcing all uploads to PyPI to 
> essentially have their long_description checked completely side steps that 
> class of problems from reoccurring.
I'm with you on this, and flit actually already checks this before upload 
(though it's a warning, rather than an error). But the difference in this case 
is that it's never an inconvenience for a downstream user - it only affects the 
person uploading to PyPI, who can presumably fix it. Enforcing things when 
installing from source affects users who can't fix any issues it highlights.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Thu, Jul 6, 2017, at 10:40 PM, Daniel Holth wrote:
> It might be more natural to pass a build directory for intermediate build 
> artefacts along with the wheel output directory to the build wheel hook. This 
> would remove pip from an awkward position of managing a copy step in the 
> middle of a build and would be more like out of tree builds in other build 
> systems. For example in automake you do out of tree builds by making a new 
> build directory and running the configure script from that directory instead 
> of the source directory. With a fresh directory old builds don't get in the 
> way.
I would also be happy with this. Though if you're trusting the backend to do a 
tidy build, do you need to pass in a directory for intermediates at all? The 
backend could just create a temporary directory itself.
I think Paul & Donald have been pretty adamantly against trusting backends to 
build tidily, though. And this certainly doesn't do anything like Donald wants 
to ensure that sdists don't omit key files.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Sat, Jul 15, 2017, at 12:54 PM, Paul Moore wrote:
> Given that this is the case we're talking
> about, my experience is that working directories contain all sorts of
> clutter - small test files I knocked up, experimental changes I
> discarded, etc. That may simply reflect the way I work, but comments
> I've seen indicate that I'm not *completely* alone. So for me, the
> point here is about making sure that "pip install ." in a cluttered
> working directory results in "what the developer wants".

If those 'clutter' files are within the package directory (i.e. next to
__init__.py), and we have to do a direct wheel build because VCS
information is not available (not a VCS checkout, or VCS not on PATH),
then I don't know how flit can avoid installing them, under any of the
proposed isolation mechanisms. We can't extract a 'pristine' source tree
from a non-pristine one without a list of what belongs in the pristine
one - which flit gets from the VCS.

distutils/setuptools mitigates this to some extent by having you
explicitly specify package_data, but between globbing and the automatic
inclusion of .py files, it's imprecise. And this is exactly the sort of
developer annoyance I want to get away from.

So I don't see a good way to avoid picking up existing clutter when you
install from source. I think the main purpose of isolating the build is
to avoid generating new clutter in the source directory.

> [1] I don't know enough about flit to be sure, but if the developer
> forgets to check in a new source file, would it be possible for that
> source file be in the wheel but not in the sdist?

For now, flit's protection against this is that it will only build an
sdist if all files in the cwd have either been added to the VCS or
ignored. So it's possible to exclude a .py file from the sdist by
vcs-ignoring it, but you can't get there just by forgetting to check in
a file.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Sat, Jul 15, 2017, at 03:28 PM, Nick Coghlan wrote:
> For flit, there are two main cases of interest:
> 
> - publisher development environments
> - building from release tarballs rather than VCS clones
> 
> In the first case, the typical scenario will have both VCS metadata
> *and* VCS tools available, so pip will successfully use the
> sdist->wheel path, and clutter will be avoided in the end result.

I agree that this will typically be true, so long as we remember that
there in a significant minority of cases in which it will not. This
includes Windows developer environments where git is used but not on
PATH, and installing from a directory bind-mounted in a docker
container.

> If flit's build_wheel *also* refuses to run in a "dirty" VCS tree

I'm reluctant for build_wheel to behave differently depending on whether
VCS metadata is available - more code, more confusion, more chances for
things to go wrong. I can see that it would reduce the likelihood of
these kinds of problem, but I suspect that it would increase the
likelihood of other kinds of problem which we haven't even discovered
yet.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Fri, Jul 14, 2017, at 11:18 AM, Nathaniel Smith wrote:
> OTOH this whole 11th hour discussion of forcing every build system to
> have in-tree and out-of-tree build support is solving some other
> problem. 

It is not my intention to force build systems to support either of
these.

- Where build systems only support out-of-tree builds, it should be
trivial to create a temporary directory when build_directory is not
passed.
- Where build systems only support in-tree builds, they should copy the
necessary files to build_directory and run an in-tree build there. This
is more complex, but it appears to be non-negotiable that there is some
way of building without affecting the source directory, so whatever the
interface is, we need some way to do this.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Thomas Kluyver]

I'm also not going to have much time to keep discussing this over the
next few days.

On Thu, Jul 20, 2017, at 11:22 AM, Paul Moore wrote:
> 2) I'd much rather if we're going for a "stop and take stock" style of
> reset, that we base that off what's currently in the official version
> of the PEP. Trying to read a new document and identify the differences
> from the working version is both time consuming and error prone. I
> don't think there's any reason that any of your questions wouldn't
> just as reasonably apply to the official version of the PEP.

Agreed. I skimmed through Nathaniel's version. The changes I notice
relative to the working version in the peps repo are:

1. Removes the build_directory= parameter
2. Removes the optional hook prepare_metadata_for_build_wheel
3. Adds an extension namespace keyed by the name of the frontend
specifying the hooks (e.g. backend.extensions['pip'].some_extra_hook )

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Sat, Jul 15, 2017, at 08:33 PM, Donald Stufft wrote:
> I wonder if maybe it would be more useful to simply recommend that instead of 
> shelling out to random vcs binaries that these projects depend on (or bundle) 
> libraries to directly interact with a repository. For instance, if your 
> project supports git, then you can use dulwich or pygit2 and then the 
> invariant of “building inside of a docker container without `git` installed” 
> still remains functional.
I did consider this kind of approach. It might be feasible for git using 
dulwich (pygit2 expects libgit2 on your system, so you can't just require it as 
a Python package). But it's ironically not workable with mercurial, even though 
it's pure Python, because hg uses Python 2, while flit requires Python 3. And I 
don't see this working reliably for svn, or bzr, or other less common VCSs.
So at least for flit, I think we will continue to rely on external, non-pip 
installable dependencies for this. This isn't a problem so long as building an 
sdist isn't necessary to get a project installed.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Thomas Kluyver]

On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote:
> Is there a prototype implementation of pep 517 yet?



- Flit has a PR with a prototype backend implementation, though it's
  not up to date with all the changes the PEP has undergone. I'll
  update it when we've agreed on a spec - it's still a fast moving
  target right now.- I have a prototype module frontends could use to call 
hooks here:
  https://github.com/takluyver/pep517 . It's mostly up to date,
  except for the issue with using sdist as a fallback for copying
  files for a wheel.
Re: magic strings - like Nathaniel said, I haven't noticed them as part of any 
proposal so far.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Tue, Jul 4, 2017, at 09:45 AM, Nick Coghlan wrote:
> +1, but we should explicitly note in the rationale section of the PEP
> that it's to cover both of the following cases:
> 
> * build from an already unpacked and potentially edited sdist"
> * cleanly support explicitly out-of-tree builds even when the
> dependencies for working with the VCS aren't available

I'm happy to add a note about these requirements. Are you planning to
merge some of Nathaniel's rewrite? If so, I'll hold off from making any
changes until that is done.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Tue, Jul 4, 2017, at 01:06 AM, Donald Stufft wrote:
> 2) We have a VCS directory or “original development source” or
>whatever you want to call the thing you have before a sdist that
>typically gets into a sdist.> - Works on both proposals for setuptools 
> and flit (since both can
>   go from a VCS to a sdist).> - Thomas might have said he’d be 
> unhappy if this case goes through
>   a real sdist… I forget the specifics of that discussion now.
Practical objection: besides it being a VCS checkout, you need the VCS
tools available (e.g. git on $PATH). It's not hard to imagine cases
where this doesn't hold, e.g. installing from a directory bind-mounted
into a docker container. Between this and your case 3 (local directory
not a VCS checkout), failures - while not common - won't be
particularly rare.
Principle objection: you don't want an sdist! You want the necessary
files copied efficiently to a clean location. You're using something
complex as a proxy for something simple.
Prediction objection: If we end up with pip asking for an sdist when
it's trying to build a wheel,  I don't want to be endlessly explaining
to people why it's broken. Nor do I want people to upload badly made
sdists because flit doesn't have the necessary information to put
extra files in there. Given the dominance of pip, I think my best
option is to find a way for build_sdist to produce an sdist which pip
accepts but pypi rejects if you try to upload it. I assume we all
agree that's not optimal?
So can we please leave the hook in place?

(We could still avoid all of this if there was a way to trust the
backend to build a wheel directly from the source directory, by the
way.)
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Tue, Jul 4, 2017, at 06:24 PM, Donald Stufft wrote:
> It occurs to me that your case here is actually a reason *not* to
> implement this hook. The goal of the hook is that the wheel built from
> the tree created by copying this file is the same as the wheel built
> from a sdist created from that same VCS directory. However if you
> require the VCS tools in order to decide what files to include in the
> sdist, then you also need those tools to decide what files to copy
> into the temporary directory. Otherwise you’ll get different outputs.
The function of the VCS is to identify extra files that are needed for
an sdist that don't affect building a wheel (like docs and tests). It's
easy for flit to identify everything necessary for building a wheel, but
those things are not sufficient for a good sdist.
> Maybe the problem boils down to the fact we’re trying to treat VCS
> directories and unpacked Sdists the same and maybe we should just add
> a file which *only* gets added to a sdist (e.g. not a development
> install or a wheel) similar to the .dist-info/WHEEL file that just
> acts as a marker for “Hey, I’m in an sdist”> 

"Hey, I'm in something which was once an sdist,  but may no longer be
clean. Or someone has copied the marker file into a directory that
was never an sdist to make something work because for some reason
there's a difference in behaviour for directories that pip thinks
come from sdists."
> 
> and if we hit that, then pip just does the copytree implementation.
> That absolves build backends from needing to worry about the unpacked
> sdist case, but makes it still function in 100% of the cases (assuming
> the build backend produces a correct sdist).
It still doesn't deal with the cases where you're not coming from an
sdist but you can't get the VCS info for whatever reason.
We had reached a compromise that we all seemed to be OK with - albeit
not very keen on. I find it very frustrating that we seem to be
rehashing the same arguments that got us there in the first place.
Thomas

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

Thank-you all for the discussion and the attempts to accommodate flit,
but I'll bow out now. It's become clear that the way flit approaches
packaging is fundamentally incompatible with the priorities other people
have for the ecosystem. Namely, I see sdists as archival artifacts to be
made approximately once per release, but the general trend is to make
them a key part of the build pipeline.

Making a guerilla tool with no concern for integration was fun. It
became frustrating as people began to use it and expected it to play
well with other tools, so I jumped on PEP 517 as a way to bring it into
the fold. That didn't work out, and a tool that doesn't play well with
pip can only be an attractive nuisance at best, even if it technically
complies with the relevant specs.

Flit is therefore deprecated, and I recommend anyone using it migrate
back to setup.py packaging.

Best wishes,
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Thu, Jul 6, 2017, at 11:55 AM, Paul Moore wrote:
> On 6 July 2017 at 11:26, Ralf Gommers  wrote:
> > I hope you'll reconsider that deprecation - flit is one of only two (AFAIK)
> > active attempts at making a saner build tool (enscons being the other one),
> > and does have real value I think.
> 
> Agreed. In spite of the fact that I've been part of the pushback
> you've had over flit's approach, I nevertheless feel that flit is a
> major step forward in providing a user-friendly project packaging tool
> for Python.

Thanks both, and Matthias. I'd reconsider it if I could see a reliable
way to support pip installing from a local directory. But at present, it
seems unavoidable that pip will require building an sdist, and I can't
see a sufficiently reliable way for flit to to do that. I compromised on
requiring a VCS to build an sdist for release, but I consider that an
unacceptable restriction for installing from source.

Flit could cheat and build a partial sdist for pip to unpack and build a
wheel from, but that becomes a problem if other tools use the hook to
generate an sdist for release.

So I see no good options for flit to be a good backend, and trying to
argue for the spec to be something I can work with is exhausting.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

Thanks Nick for the detailed reply. I have read it carefully, and you've
probably convinced me to get back on board. Some more responses inline:

On Thu, Jul 6, 2017, at 03:38 PM, Nick Coghlan wrote:
> While I can completely understand how the current debate over whether
> or not the prepare_input_for_build_wheel hook is necessary or not
> would make you feel that way, I hope I can convince you that we're
> really just quibbling over a genuinely trivial arcane technical detail
> that I'd never let get in the way of flit being a full-fledged
> participant in the Python packaging ecosystem.

To be clear, I don't particularly care for the hook. I can see that it's
something of a kludge between two competing approaches.

What is important to me is that if a user installs from source the
obvious way (pip install . ), failure to build an sdist does not result
in a failure to install. The extra hook was one approach to that, but
it's also OK by me if it tries to make an sdist and falls back to either
copytree or an inplace build.

> That is, the current point of contention is specifically about how we
> want tools to behave when we're starting with a source directory that:
> 
> 1. Doesn't include VCS metadata (e.g. it's been exported as a tarball
> rather than cloned)
> 2. The build frontend doesn't want to use as the basis for an in-place
> build
> 3. The build frontend doesn't want to blindly copy into a separate
> build directory
> 
> So just by way of those preconditions, we're already well outside the
> most common package installation workflows.

One of my concerns in this debate is that this is presented as a very
rare corner case that we don't have to worry about too much. I agree
that it's not the most common case, but I think it's common enough that
we should care about making it easy,  given that:

- Condition 1 also covers directories with VCS metadata where the VCS
tools are not on $PATH. Another case occurred to me recently: Windows
users who have installed git but not added it to the default PATH.
- Conditions 2 and 3 seem likely to be the default for a source install
with pip.

As an order of magnitude, I'd estimate this is ~10% of installs from a
source directory - which is to say, moderately common.

> That perspective is embodied in the hypothetical proposal to add a
> "--build-strategy" option to pip that would allow folks building
> wheels to choose between:
> 
> - creating and unpacking an sdist and building a wheel from that
> - copying the directory tree and building a wheel from that
> - building a wheel directly from the original directory
> 
> (Perhaps with a variant that tries to create and unpack the sdist
> first, and only if that fails falls back to copying the entire tree)

This could be useful flexibility for advanced users. But I worry that
pip will use the 'sdist' build strategy by default, and expect users to
handle cases where that fails. I think this would be a mistake. From a
user perspective, it would mean:

- "pip install ." is the recommended way to install from source, but in
some situations it doesn't work.
- Adding the mystic incantation "--build-strategy direct" makes it work,
and from a user perspective makes absolutely no difference to the
result.

Of course, I also have a vested interest in things not working this way:
I would get a steady trickle of people asking "why does flit require a
VCS to install from source?" From my perspective, it doesn't require
that, but I would be unable to 'fix' it.

Donald:
> I think it is a complete non-starter to suggest removing installation from 
> sdist support from pip

I'm certainly not suggesting that (hopefully this was already clear, but
just in case ;-)

> the question then becomes do we want to try and push things towards only 
> having *one* primary flow through the state machine of Python’s packaging, or 
> do we want to support transitions that allow you to “skip” steps. 

My idealised view of the state machine is something like this:

wheel <-- source tree <--> sdist

I agree that there's a problem with losing important data when you go
[source tree --> sdist --> source tree] - in fact this is one of the
pain points I was trying to avoid with flit. But I don't like the idea
of solving that by saying that all wheels must have passed through an
sdist; it feels like a redundant there-and-back-again journey.

So how else could we tackle the systematic problem? It's definitely a
good idea to ensure that [stree --> sdist --> stree --> wheel] doesn't
miss out anything that [stree --> wheel] includes, but I'd focus on
doing this in developer tools, e.g.:

1. Tools such as flit could check it when you're building a release
2. Tools running on CI services could build both and compare them
3. Bots could scan PyPI for projects with both a .whl and a .tar.gz,
build a wheel from the tarball, compare them, and notify the maintainer
if there's a problem.

In the short term, I reckon that 2 is the most promising - we can make a

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Wed, Jul 5, 2017, at 05:08 PM, Paul Moore wrote:
> is that flit doesn't handle scenarios like "I unpacked a sdist" or "I
> downloaded the project archive from github and unpacked that" well.

Flit handles these fine for everything *apart* from making an sdist. It
can make a wheel, install the package, or symlink it as a development
install. Hence why I'm so frustrated by the insistence that we must make
an sdist when we have no need for an sdist.

It's not a compromise I'm entirely happy with, but all the other options
that we came up with had bigger problems, IMO.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Provide separate development and documentation URLs in PyPI metadata?

[Thomas Kluyver]

On Tue, Jun 27, 2017, at 04:58 PM, Barry Warsaw wrote:
> On Jun 25, 2017, at 11:33 AM, Nick Coghlan wrote:
> >I'd favour "Participate" over any variant of "Contribute", as without
> >context, "Contribute" makes me think of financial support in the
> >crowdfunding/tip jar sense.
> 
> "Participate" may mean two different things.
> 
> * Here's the development home, with a repo and issue tracker,
> contributions
>   welcome!
> 
> * Here's the mailing list or other forum where we discuss the future of
>   Guido's Magical Mystery Time Machine.

Perhaps this points to labelling URLs with nouns rather than verbs:
things like 'mailing list', 'source code' or 'issue tracker' seem less
ambiguous than 'participate' or 'contribute'.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Thu, Jul 6, 2017, at 11:51 PM, Paul Moore wrote:
> On reflection, I'm less concerned about this than I was. If you wanted
> to propose a stripped down version of PEP 517 which assumed it was the
> backend's responsibility to ensure reproducible isolated builds, I'd
> be willing to listen. But the proposal would need to include some
> pretty strong requirements on precisely what we're asking of backends
> - if build isolation is pip's problem to solve, then I'm happy for us
> (the pip devs) to take that responsibility, and agree hooks that we
> need to do so, but if we're assuming backends handle it for us, I
> think we need to document clearly what we're assuming (because
> frankly, the pip devs are the ones with the experience of the
> potential issues).

How does this sound to you:

"""
If build_directory is not None, it is a unicode string containing the
path to a directory where intermediate build artifacts may be stored.
This may be empty, or it may contain artifacts from a previous build to
be used as a cache. The backend is responsible for determining whether
any cached artifacts are outdated. When a build_directory is provided,
the backend should not create or modify any files in the source
directory (the working directory where the hook is called). If the
backend cannot reliably avoid modifying the directory it builds from, it
should copy any files it needs to build_directory and perform the build
there.

If build_directory is None, the backend may do an 'in place' build which
modifies the source directory. The semantics of this are not specified
here.

In either case, the backend may also store intermediates in other cache
locations or temporary directories, which it is responsible for
managing. The presence or absence of any caches should not make a
material difference to the final result of the build.
"""

> this is a case where flit is
> *not* a good example of a backend, as it avoids the nasty cases by
> design).

Agreed. Enscons is a better example, though, and Daniel seems confident
that out-of-tree builds are feasible.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Thomas Kluyver]

On Wed, Aug 2, 2017, at 01:05 PM, Nick Coghlan wrote:
> I wouldn't say that, as the two most clearly side effect free ways of
> putting the source directory on sys.path are:
> 
> 1. make it the current working directory for a "python -m" or "python
> -c" invocation, so that the current directory gets injected as
> sys.path[0] by the interpreter. After all, it's already a requirement
> in the PEP that the current working directory be the directory
> containing pyproject.toml, so this approach handles that as well.
> 2. run a backend script directly (so it's the script directory that
> gets implicitly added to sys.path by the interpreter), and then inject
> the current working directory as sys.path[0] before importing the
> nominated backend
> 
> If a frontend decides to go messing about with PYTHONPATH instead of
> doing either of those, then that's on them (and after it breaks
> horribly, they'll hopefully realise the error of their ways and switch
> to one of the other more reliable approaches).

Nathaniel, can you live with the source directory being on sys.path to
load the hooks? Or would you like to have another go at convincing
people that it shouldn't be?

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Thomas Kluyver]

On Fri, Jul 28, 2017, at 09:37 PM, Daniel Holth wrote:
> See how trivial it would be to put the delegated sdist generator into the 
> build-backend within the confines of the current PEP?
I agree that delegation within the backend is a reasonable answer to this.
> The build-backend could point to a .py in the current directory that 
> implements itself with different tools,
I think this was actually a question that someone (Nathaniel?) brought up: is 
the project source directory on sys.path when the backend is loaded? I would 
lean yes, so that it's possible to have a custom backend in the project tree, 
but I think whoever brought it up before was leaning no.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Thomas Kluyver]

On Fri, Jul 28, 2017, at 04:16 PM, Daniel Holth wrote:
> It looks like we've run out of things to say about PEP 517, except, how soon 
> can we get it into pip?
I admire your optimism! ;-)

While I partly hope that I get a unanimous disagreement, as it would be 
simpler, I have a nagging concern about something that someone mentioned ages 
ago: does it make sense for building sdists and building wheels to be part of 
the same backend?
Flit now makes both sdists and wheels, but for a long time it only made wheels, 
and the two parts are largely separate: it wouldn't take much work to use 
flit's sdist machinery but build the wheel with a different tool (e.g. if it 
had compiled parts).
Requiring one backend to build both formats may result in a significantly 
higher barrier to entry for backend developers: I don't know if I would have 
started writing flit if PEP 517 had already been finalised and I had to make 
both wheels and sdists to comply with it. They're also, at least to my mind, 
quite different kinds of thing: an sdist is almost like an archive of a VCS 
tag, whereas a wheel is the end result of any build steps the project needs.
So I'd like us to circle back round and reconsider allowing projects to specify 
'use tool X to make wheels, and tool Y to make sdists'. If everyone else thinks 
that's unnecessary, I think we'd all be glad to finish this discussion up, but 
this concern has been growing in my mind for a while, and I want to get it out 
there before we finalise the PEP.
Thomas

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Thomas Kluyver]

Are we content to say that sys.path includes the source directory where
the hook is run? Shall I prepare a PR against the PEP for that?

On Sun, Jul 30, 2017, at 02:12 PM, Nick Coghlan wrote:
> On 30 July 2017 at 02:46, Nathaniel Smith  wrote:
> > Or am I worrying about a non-issue and it's fine if flit imports click from
> > the source tree?
> 
> Don't worry about it too much, as the problem here isn't really any
> worse than it is for normal runtime dependencies of any other project
> that relies on having the current directory first in sys.path. It just
> so happens that the project in question in this case is a Python
> project's build system.
> 
> Due to the preference for a flat module namespace as the default,
> there are plenty of ways to hit name shadowing problems in Python, and
> as Donald notes, build systems have other motives to vendor their
> dependencies rather than installing them normally.
> 
> Just switching the path order as has been suggested also doesn't solve
> the problem, as it merely inverts the issue: having "some_name"
> installed in site-packages would break source installations of
> packages that expected to be able to import "some_name" from their own
> root directory.
> 
> If the problem does come up in practice, then there are a number of
> ways for affected projects to work around it in their project
> directory structure:
> 
> 1. Use a top-level "src" directory (we may want to reserve "src" on PyPI)
> 2. Use a top-level "tools" directory (we may want to reserve "tools" on
> PyPI)
> 3. Add a leading or trailing underscore to the local directory name
> (as while that's legal for Python imports, it's prohibited for PyPI
> project names, and hence will often sidestep naming conflicts with
> published packages)
> 
> Beyond that, the only approaches I'm aware of that systematically
> avoid this kind of problem at the language design level are to either
> use URL-based imports (ala Java or Go), or else to have separate
> syntax for "system-only" and "local resolution permitted" imports (ala
> C and C++), and Guido opted not to pursue either of those strategies
> for Python.
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build frontend responsibilities

[Thomas Kluyver]

I'll let Nathaniel relay them.


On Fri, Aug 18, 2017, at 06:23 PM, xoviat wrote:
> Thomas:
> 
> 
> What are the specific issues that need to be worked out? 
> 
> 
> Regards,
> 
> 
> xoviat
> 
> 2017-08-18 3:09 GMT-05:00 Thomas Kluyver <tho...@kluyver.me.uk>:
>> On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote:
>>  > The other aspect I'm not clear on is whether or not both PEP
>>  > authors are otherwise happy for the current version to be
>>  > *considered* for acceptance
>>
>> Nathaniel has mentioned to me a list of issues he sees that we still>>  need 
>> to work through. So I think we're going to be working on
>>  this for a>>  while yet.
>> 
>>  Thomas
>> ___
>>  Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build frontend responsibilities

[Thomas Kluyver]

There's prototype machinery to call the hooks here, though it's outdated
because the spec keeps changing:https://github.com/takluyver/pep517

There's a prototype backend in a branch of flit, but it's even
more outdated:https://github.com/takluyver/flit/tree/toml-config


On Fri, Aug 18, 2017, at 06:27 PM, Daniel Holth wrote:
> Apart from the issues, can we get some prototype implementations?


> 
> On Fri, Aug 18, 2017, 13:24 xoviat <xov...@gmail.com> wrote:
>> Thomas:
>> 
>> 
>> What are the specific issues that need to be worked out? 
>> 
>> 
>> Regards,
>> 
>> 
>> xoviat
>> 
>> 2017-08-18 3:09 GMT-05:00 Thomas Kluyver <tho...@kluyver.me.uk>:
>>> On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote:
>>>  > The other aspect I'm not clear on is whether or not both PEP
>>>  > authors are otherwise happy for the current version to be
>>>  > *considered* for acceptance
>>>
>>> Nathaniel has mentioned to me a list of issues he sees that we still>>>  
>>> need to work through. So I think we're going to be working on this
>>>  for a>>>  while yet.
>>> 
>>>  Thomas
>>> ___
>>>  Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>> 
>> ___
>>  Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build frontend responsibilities

[Thomas Kluyver]

I wouldn't expect established tools like pip & setuptools to accept pull
requests implementing a PEP which is still under discussion and subject
to change. They will naturally want a stable, accepted spec before they
consider merging code for it.
We've probably all wished that the discussion could be brought to a
swift conclusion. But there are real questions to work out, and people
have many other things to pay attention to. I'm frustrated by how long
it's taking as well, but there's no magic button anyone can press to
make it go quickly.
Thomas




On Fri, Aug 18, 2017, at 08:17 PM, xoviat wrote:
> At this point, I can only offer implementations as I have been
> attempting to do (some of which are outdated because the PEP has
> changed), but I cannot make anyone pay attention to or accept my pull
> requests and it seems I cannot accelerate the discussion here.> 
> 2017-08-18 14:08 GMT-05:00 xoviat <xov...@gmail.com>:
>> I have attempted to implement prototype machinery here[1] and
>> here[2], but it's been widely ignored IHMO because as Thomas said,
>> the spec keeps changing.>> 
>> 2017-08-18 12:37 GMT-05:00 Thomas Kluyver <tho...@kluyver.me.uk>:
>>> __
>>> There's prototype machinery to call the hooks here, though it's
>>> outdated because the spec keeps changing:>>> 
>>> https://github.com/takluyver/pep517
>>> 
>>> There's a prototype backend in a branch of flit, but it's even more
>>> outdated:>>> https://github.com/takluyver/flit/tree/toml-config
>>> 
>>> 
>>> On Fri, Aug 18, 2017, at 06:27 PM, Daniel Holth wrote:
>>>> Apart from the issues, can we get some prototype implementations?>>>> 
>>>> On Fri, Aug 18, 2017, 13:24 xoviat <xov...@gmail.com> wrote:
>>>>> Thomas:
>>>>> 
>>>>> 
>>>>> What are the specific issues that need to be worked out? 
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> 
>>>>> xoviat
>>>>> 
>>>>> 2017-08-18 3:09 GMT-05:00 Thomas Kluyver <tho...@kluyver.me.uk>:
>>>>>> On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote:
>>>>>>  > The other aspect I'm not clear on is whether or not both PEP
>>>>>>  > authors are otherwise happy for the current version to be
>>>>>>  > *considered* for acceptance
>>>>>>
>>>>>> Nathaniel has mentioned to me a list of issues he sees that we
>>>>>> still>>>>>> need to work through. So I think we're going to be working on
>>>>>> this for a>>>>>> while yet.
>>>>>> 
>>>>>> Thomas
>>>>>> ___
>>>>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>>>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>>> 
>>>>> ___
>>>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>> 


Links:

  1. https://github.com/pypa/pip/pull/4589
  2. https://github.com/pypa/setuptools/pull/1039
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Bootstrapping setuptools

[Thomas Kluyver]

On Wed, Aug 23, 2017, at 05:37 AM, Nick Coghlan wrote:
> On 23 August 2017 at 05:24, xoviat  wrote:
> > The question that I have is: how does flit address this issue? Does flit
> > require external dependencies to bootstrap itself?
> 
> It does, but those dependencies currently use setuptools as their
> packaging system rather than flit.

Yup. I actually moved a package which I maintain (zipfile36) back to
setuptools packaging specifically so that flit could use it without a
dependency loop.

The other dependencies for bootstrapping are requests and docutils.
There's one other dependency listed, but that can be ignored for
bootstrapping.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build frontend responsibilities

[Thomas Kluyver]

On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote:
> The other aspect I'm not clear on is whether or not both PEP authors
> are otherwise happy for the current version to be *considered* for
> acceptance

Nathaniel has mentioned to me a list of issues he sees that we still
need to work through. So I think we're going to be working on this for a
while yet.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Open questions around artifact export directories

[Thomas Kluyver]

On Mon, Jun 12, 2017, at 10:09 PM, Donald Stufft wrote:
> It’s pretty hard to screw up zipping up a directory.

If you want reproducible builds, it's very easy to screw up, and your
response doesn't inspire confidence that frontends will do it carefully.
But I see flit mainly as something you use directly to build and publish
wheels, and these hooks of secondary interest for e.g. installing from a
VCS URL. So I'll keep my zip-archive-building code in flit, and let
frontend tools duplicate the part of that functionality they care about.
It's also, of course, hard to screw up unzipping to a directory. ;-)


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Open questions around artifact export directories

[Thomas Kluyver]

On Mon, Jun 12, 2017, at 09:45 PM, Daniel Holth wrote:
> I think all my wheel generators except bdist_wheel build the zipfile
> directly.
There is a certain appeal to using the zipped .whl file as the canonical
format for all tools that produce or consume wheels, rather than
defining a closely related but distinct 'unpacked wheel' format. A
directory and a zip file do not have 100% identical features (filename
encodings may differ, entries in a zip file are ordered, there may be
metadata in one format that's not present in the other, and so on).
I think we're also making this change in the assumption that
frontends will be few and backends numerous, so it makes sense to
shift more work into frontends. That may not necessarily be true - I
could imagine more frontends emerging while people standardise on
just a few backends. Jupyter's frontend/kernel separation was
initially designed in the belief that it would support one kernel and
many frontends, but we've ended up getting a lot of kernels with just
a handful of popular backends.
I don't feel strongly about this - I can build a wheel and then unzip it
again if that's what the spec says. But given the choice, I'd specify it
in terms of a zipped .whl file rather than a directory.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Open questions around artifact export directories

[Thomas Kluyver]

On Mon, Jun 12, 2017, at 10:23 PM, Donald Stufft wrote:
> because it allows the front ends more flexibility in how they use
> the wheels
I don't get this? Why is it more flexible?

> and allows us to avoid doing work, making the process involved faster
> for everyone.
This is true so long as backends build a directory and then skip zipping
it up. If backends are building a zip file and then unpacking it (which,
from what Daniel and I have described, may be common), then for tasks
which want a zip file, you're now unpacking and repacking it.
So it hinges both on what backends do and on what tasks are common for
frontends using this interface. You might assume that the most common
task will be installation, which uses the files unpacked. But most
installs will use a pre-built wheel, so it's not obvious to me that the
typical use of the build interface will be to install a package.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Open questions around artifact export directories

[Thomas Kluyver]

On Sat, Jun 10, 2017, at 06:14 PM, Nick Coghlan wrote:
> Thomas - I agree with Donald's reasoning here, so would you mind
> updating the PEP accordingly?

I've done so here:
https://github.com/python/peps/pull/290

There are still a couple of questions on which I wasn't quite sure what
the consensus is:

-Do we want to rename the build_wheel hook now that it makes an
unpacked wheel, e.g. export_wheel_contents to match
export_sdist_contents?
-I have assumed that the wheel hook puts its contents in the
directory it's passed, rather than creating a subfolder. This is in
keeping with the structure of wheels, which do not have a single
top-level directory (unlike sdists), but it wouldn't fit with a future
hypothetical extension to build multiple wheels at once; we would need a
separate hook for that.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI

[Thomas Kluyver]

For reference, I switched the default upload server for flit in version
0.11.

On Sat, Jun 24, 2017, at 08:45 AM, Nick Coghlan wrote:
> On 24 June 2017 at 15:24, Nick Coghlan  wrote:
> > https://packaging.python.org/guides/tool-recommendations/ is missing a
> > section on "Publishing platform" that explains the migration that is
> > currently in progress and covers what pypi.org currently handles
> > better than pypi.python.org (publishing), which should still use
> > pypi.python.org (downloading due to server bandwidth availability,
> > package management UI), and those where it doesn't matter much which
> > you use (browsing).
> 
> The new "Publishing Platform Migration" section is now live:
> https://packaging.python.org/guides/tool-recommendations/#publishing-platform-migration
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Thomas Kluyver]

I have prepared a PR against the PEP adding get_build_sdist_requires ,
and renaming a couple of the other hooks for clarity
(get_build_wheel_requires, prepare_build_wheel_files):

https://github.com/python/peps/pull/297
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Provide separate development and documentation URLs in PyPI metadata?

[Thomas Kluyver]

On Sat, Jun 24, 2017, at 06:34 PM, Brett Cannon wrote:
> Anyway, just an idea I had based on my flit experience yesterday plus
> a tweet sent to me. (And if PyPI already supports this somehow then
> Thomas should brace for the feature request from me .)
This prompted me to go and look at the metadata PEPs. I thought the URL
fields were only 'Home-Page' and 'Download-URL', but in fact PEP 345
added a multi-use 'Project-URL' field:
https://www.python.org/dev/peps/pep-0345/#project-url-multiple-use

I quite like this idea - rather than prescribing a couple of specific
kinds of URL, it lets the project author list as many as make sense.
Perhaps we should recommend a set of common labels for URLs in this
field, e.g. 'Documentation', 'Bug tracker', 'Source repository'.
This does leave open the question of which one to put in the mandatory
'Home-Page' field (I usually use the address of the repo), and whether
to duplicate it in a 'Project-URL'.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Thomas Kluyver]

On Sun, Jun 25, 2017, at 08:41 AM, Nick Coghlan wrote:
> Aside from not producing an archive, the bit that makes
> `prepare_wheel_build_files` notably different from `build_sdist` is
> that it relies on `get_build_wheel_requires` *not* on
> `get_build_sdist_requires`:
> https://www.python.org/dev/peps/pep-0517/#build-environment

This reminds me: while implementing a wrapper to call PEP 517 hooks, a
problem occurred to me. I have implemented prepare_wheel_build_files
with a fallback to making an sdist if the hook is not defined:

https://github.com/takluyver/pep517/blob/ee43a9334c377d7c37badcc8527cb7a8500180f7/pep517/_in_process.py#L136

But in this case, the build_sdist hook will be invoked in an environment
with the build-wheel-deps installed, not the build-sdist-deps. Working
around this would require moving the fallback a couple of levels up the
stack to a component that knows about installing packages for the build
process. That's not impossible, but it's inelegant and less efficient.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Thomas Kluyver]

Nick has merged that PR, and the updated PEP is visible here:

https://www.python.org/dev/peps/pep-0517/

Hopefully we're nearing a consensus on this now. If you're interested,
please do have a read through the latest version.

Thomas

On Sat, Jun 24, 2017, at 02:44 PM, Thomas Kluyver wrote:
> I have prepared a PR against the PEP adding get_build_sdist_requires ,
> and renaming a couple of the other hooks for clarity
> (get_build_wheel_requires, prepare_build_wheel_files):
> 
> https://github.com/python/peps/pull/297
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

On Sat, May 20, 2017, at 07:54 AM, Nick Coghlan wrote:
> * on platforms with 8-bit standard streams (e.g. Linux, Mac OS X),
> build systems SHOULD emit UTF-8 encoded output
> * on platforms with 16-bit standard streams (e.g. Windows), build
> systems SHOULD emit UTF-16-LE encoded output

I'm quite prepared to accept that I'm mistaken, but my understanding is
that *standard streams* are 8-bit on Windows as well. The 16-bit thing
that Python 3.6 does, as I understand it, is to bypass standard streams
when it detects that they're connected to a console, and use a Windows
API call to write text to the console directly as UTF-16.

If so, when stdout/stderr are pipes, which I assume is how pip captures
the output from build processes, there's no particular reason to send
UTF-16 data just because it's Windows.

Thomas

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

On Sat, May 20, 2017, at 05:36 PM, Steve Dower wrote:
> I'll probably read the PEP closely and see that this is entirely 
> incorrect, but if it's right:
> 
> * encoding for text between the build UI and build tool should just be 
> specified once for all platforms (i.e. use UTF-8).

+1

> * encoding for text between build tool and the compiler depends on the 
> compiler
> 
> In general, since most subprocesses (at least on Windows) do not have 
> customizable encodings, the tool that launches them needs to know what 
> the encoding is. Since we don't live in a Python 3.6 world quite yet, 
> that means the tool should read raw bytes from the compiler and encode 
> them to UTF-8.

I half agree, but:
- Build tools may not 100% know what encoding output will be produced,
especially if the developer can supply a custom command for the build
tool to run.
- It's possible for data on a pipe to be binary data with no meaning as
text.
- As a lazy developer, I don't want to read stdout/stderr from a
subprocess only to spit it back to my own stdout/stderr. I'd much rather
just launch the subprocess and let it use the same stdout/stderr as my
build tool.

So I think it's most practical to recommend that build tools produce
UTF-8 (if not sys.stdout.isatty()), but let build tool developers decide
how far they go to comply with that.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Introducing PyPIContents

[Thomas Kluyver]

Hi Luis,

Awesome, thanks for this :-). It was me posting before about
indexing PyPI.
I'm intrigued: how do you keep it up to date using Travis? When I
looked into this, I was pretty sure you need to download every package
to index it. Do you have some way to only download the new releases? Or
is Travis able to download every package every day? Or have you found
another way round it?
Does the index only include the latest version of each package, or does
it also include older versions? The wifi on the train I'm on at the
moment isn't fast enough to download 60 MB to find out. ;-)
Does your indexing tool prefer to use wheels or sdists? Is it capable of
using either for packages which don't have both available? Do you do
anything to cope with modules which may be included for one platform but
not another?
I'm excited to see someone actually doing this!

Thomas


On Sat, May 20, 2017, at 03:01 AM, Luis Alejandro Martínez Faneyth wrote:> Hi 
everyone,
> 
> I'm new to this list but I've been reading some threads in the
> archive.> 
> Around february, an idea about indexing modules from PyPI packages was
> brought up. I've been working on something similar for quite a while.> 
> PyPIContents is an index of PyPI packages that lists its modules and
> command line scripts in JSON format, like this:> 
> 
> [
> ..."1337": { "cmdline": [], "modules": [ "1337", "1337.1337" ],
> "version": "1.0.0" }, ... ]
>
>> You can check it out here:
> 
> https://github.com/LuisAlejandro/pypicontents
> 
> And some use cases:
> 
> https://github.com/LuisAlejandro/pypicontents#use-cases
> 
> The actual index lives here, its around 60MB:
> 
> https://raw.githubusercontent.com/LuisAlejandro/pypicontents/contents/pypi.json>
>  
> Is updated daily with the help of Travis:
> 
> https://github.com/LuisAlejandro/pypicontents/blob/contents/.travis.yml> 
> Anyway, I hope is useful and I'll be around for any comments or
> questions.> 
> Cheers!
> 
> 
> 
> Luis Alejandro Martínez Faneyth
> Blog: http://huntingbears.com.ve
> Github: http://github.com/LuisAlejandro
> Twitter: http://twitter.com/LuisAlejandro
> 
> CODE IS POETRY
> 
> _
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

On Tue, May 23, 2017, at 09:08 AM, Paul Moore wrote:
> I strongly
> prefer using the locale encoding as the assumed encoding for the
> output stream rather than UTF-8.

I may have missed it, but has anyone proposed what it should do if it
wants to send characters which can't be encoded in the locale encoding?
Paths on Windows are handled natively as UTF-16, as I understand it, so
it's entirely possible for them to contain characters which can't be
represented in, say, CP1252.

Given this, and the workarounds Nick has pointed out are necessary for
systems where the locale thinks it's ASCII, I still think that
specifying "UTF-8" is a better option than trying to work with locale
encodings. We're building a new spec for new tools in 2017, let's not
prolong the pain of platform-dependent default encodings further.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

On Tue, May 23, 2017, at 11:04 AM, Paul Moore wrote:
> However, if we do this then we have a situation where existing build
> tools (compilers, etc) that we have to support still use platform
> dependent encodings. That's a reality that we can't wish away. And the
> majority of real-life issues reported on pip are with compilation
> errors. So do we require backends that run these tools to ensure that
> they transcode the output, or do we risk significant output
> corruption, because (essentially) every high-bit character in the
> compiler output will be replaced as it's invalid UTF-8?

As you described earlier, though, even using a locale dependent encoding
doesn't really avoid this problem, because of tools using OEM vs ANSI
codepages on Windows. And if PYTHONIOENCODING is set, Python processes
will use that over the locale encoding. I think we're ultimately better
off specifying a consistent encoding rather than trying to guess about
it.

I'm also thinking of all the bugs I've seen (and written) by assuming
open() in text mode defaults to UTF-8 encoding - as it does on the Linux
and Mac computers many open source developers use, but not on Windows,
nor in all Linux configurations.

So I'd recommend that backends running processes for which they know the
encoding should transcode it to UTF-8. I expect we can make standard
utility functions to wait for a subprocess to finish while reading,
transcoding, and repeating its output.

I'm still not sure what the backend should do when it runs something for
which it doesn't know the output encoding. The possibilities are either:

- Take a best guess and transcode it to UTF-8, which may risk losing
some information, but keeps the output as valid UTF-8
- Pass through the raw bytes, ensuring that no information is lost, but
leaving it up to the frontend/user to deal with that.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

On Tue, May 23, 2017, at 12:56 PM, Paul Moore wrote:
> So based on your proposal, won't you introduce similar bugs by using
> print() without sorting out encodings? Unless (see below) you assume
> that the frontend sorts it out for you.

If you strictly follow the locale encoding, you need to sort it out in
Python anyway, in case the stdout encoding has been overridden by
PYTHONIOENCODING, or PYTHONSTARTUP, or the infernal .pth files. I accept
that those are corner cases, though.

> Yes, subprocesses that produce a known encoding are trivial to deal
> with. But remembering that you *need* to deal with them less so. My
> concern here is the same one as you quote above - assuming that
> subprocess returns UTF-8 encoded bytes, because it does on Linux and
> Mac.

I agree, that is a concern.

> But if you genuinely don't know (or worse, know there is no consistent
> encoding) I'm not sure I see how passing unknown bytes onto the
> frontend, which by necessity has less context to guess what those
> bytes might mean, is the right answer. The frontend is better able to
> know what it wants to *do* with those bytes, but "convert them to text
> for the user to see" is the only real answer here IMO (sure, dumping
> the raw bytes to a file might be an option, but I imagine it'll be a
> relatively uncommon choice).

I was indeed thinking of dumping them to a file. It's not very user
friendly, but it means the information is there if you need it. I
suspect that regardless of the locale, technical information like code
and filesystem paths will often contain enough ASCII that a human can
interpret them even if non-ASCII characters are wrongly encoded. So I
hope that needing to reverse-engineer the encoding will be relatively
rare.

The appeal of this is that it follows "in the face of ambiguity, refuse
the temptation to guess". If the backend guesses the encoding
incorrectly, the frontend gets valid UTF-8, but is no better able to
display it meaningfully, and you then need to go through
decode-encode-decode to recover the original text, even if no data was
lost.

Another option: if the backend runs a subprocess with unknown output
encoding, it redirects that output to a temp file and prints the path in
its own output. Then there's a better chance that the unknown encoding
is at least consistent within the file, so tools can do encoding
detection on it.

> At the end of the day, there is no perfect answer here. Someone is
> going to have to make a judgement call, and as the PEP author, I guess
> that's you. So at this point I'll stop badgering you and leave it up
> to you to decide what the consensus is. Thanks for listening to my
> points, though.

I know what I think, but I don't feel like there's a consensus as yet.

Can I take a quick poll of what people following this topic think?

Q1: Default encoding for captured build stdout/stderr
a. UTF-8 (consistent, can represent any character)
b. Locale default (convenient if backend runs subprocesses which produce
output in the locale encoding)

Q2: Handling unknown encodings from subprocesses
a. Backend should ensure all output is valid in the target encoding
(Q1), though it may not be accurate.
b. Unknown output may be passed on as bytes without transcoding, so the
frontend can e.g. dump it to a file.

I'm currently 1:a, 2:?a .

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

On Tue, May 23, 2017, at 04:20 PM, Nick Coghlan wrote:
> Up to this point, I've been in favour of both 1b and 2b, since they're

Noted.

> However, I also realised that there's a potential third way to handle
> this problem: design a Python level API that allows front ends to use
> more structured data formats (e.g. JSON) for communication between the
> frontend and their backend shim.
> 
> In particular, I'm thinking we could move the current
> "config_settings" dict onto something like a "build context" object
> that, *even in Python 2*, offers a Unicode "outstream" and
> "errstream", which the backend is then expected to use rather than
> writing to sys.stdout/err directly. That context could also provide a
> Python 3 style "run()" API for subprocess invocation that implemented
> the preferred stream handling behaviour for subprocess invocation
> (including applying the "backslashreplace" error handler regardless of
> version)

I'm not really compelled by this so far:

- It's more complexity for build tools - instead of just producing
output as usual, now they have to pass around a context object and
direct output to it.
- What does the frontend do if there is output on stdout/stderr anyway?
Throw it away? Let it go straight to the terminal? Reprimand the backend
for not using the streams in the build context? Or try to include it as
part of build output anyway?
- I don't see how it solves the issue with subprocesses producing
unknown encodings. The output bytes still need to be interpreted
somehow.

I'll propose a variant of an idea I described already: the frontend
could provide the backend with a fresh temp directory. If the backend
needs to run other processes, it can redirect the output into a file in
that temp directory. Then you have files with an unknown encoding,  but
each file will hopefully have one encoding, and you can use a tool like
chardet to guess what it is.

Thomas

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

On Wed, May 24, 2017, at 01:22 AM, Chris Jerdonek wrote:
> 1) Would it make sense to provide a way for build tools to specify
> what encoding they use (e.g. if not using the default), instead of
> changing their encoding to conform to a standard? It seems like that
> could be easier, although I know this doesn't address problems like
> non-conforming tools.

Interesting idea, but I'm not convinced it actually makes anything
easier. You still have the same issues if the backend runs a subprocess
which doesn't produce output in the expected encoding. And there would
be some small amount of added complexity to communicate the encoding to
the frontend.

> 2) In terms of debugging, in cases where there are encoding-related
> errors, it would help if the overall system made it easy to pinpoint
> which parts of the system are at fault (using good error handling,
> diagnostic messages, etc).

Agreed.

Nick:
> That's actually pretty similar to the way tools like mock (the chroot
> based RPM builder) work. That way, build backends could choose
> between:
> 
> - use pipes to stream output from the tools they call, deal with
> encoding issues themselves
> - redirect output to a suitable named file in the tool log directory

Do you know if that system works well for mock? Shall I try to draft a
spec of something like this for PEP 517?

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

Hi,

I'd like to make another push for PEP 517, which would make it possible
to build wheels from a source tree with other build tools, without
needing setup.py.

https://www.python.org/dev/peps/pep-0517/

Last time this was discussed we made a couple of minor changes to the
PEP, but we didn't want to accept another packaging related PEP until
PEP 518 was implemented in pip. I'm pleased to say that that
implementation has just been merged:
https://github.com/pypa/pip/pull/4144 .

I have an implementation of a build backend according to the current PEP
517 spec in a pull request on flit:
https://github.com/takluyver/flit/pull/77/files#diff-6df5630df9568322a453f03159fc8fe8

Thanks,
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Introducing PyPIContents

[Thomas Kluyver]

On Sat, May 20, 2017, at 07:29 PM, Luis Alejandro Martínez Faneyth wrote:> It 
supports ['.whl', '.egg', '.zip', '.tgz', '.tar.gz', '.tar.bz2']
> formats, and it extracts the data using any available.
Nice! If there are multiple of those formats present, does it get the
data from just one? Or does it get data from all of them and combine
it somehow?
> 
> I wasn't aware of the fact that some modules may be on one platform
> and not in another. I guess there's room for improvement.
It probably doesn't matter for most cases, but since setup.py runs
arbitrary code, it's possible for it to install different modules in
different situations - or even select modules at random, if you really
want to confuse tools like this. ;-)
This is why my own efforts at indexing focused on wheels - you can be
sure of exactly what a wheel contains. My wheel-indexing tool 'wheeldex'
is here, if there's any code or ideas there that you can 
use:https://github.com/takluyver/wheeldex

> Thank you. I made this because I wanted to have an app that guessed
> python dependencies from code by scaning module imports and then
> looking up the Index. That app is called Pip Sala Bim and you can
> check it out here:> 
> https://github.com/LuisAlejandro/pipsalabim

Neat, that's precisely one of the use cases I was thinking of for an
index. The other thing I'm interested in is providing an interface to
install modules by their import name rather than their PyPI name; I
think your index should work for that as well. I'll dig into the code of
both PyPIContents and Pip Sala Bim more soon.
Thanks,
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

On Sat, May 20, 2017, at 07:36 PM, Steve Dower wrote:
> Require that build tools either send UTF-8 to the UI component, or write 
> bytes to a file and call it a build output. I see no benefit in 
> requiring both the build tool and the UI tool to guess what the text 
> encoding is.

I'm not proposing that the install tool should try to guess the
encoding, but I think a well written install tool shouldn't crash if the
build output doesn't match the encoding it expects. Even if the spec
says that the build output MUST be UTF-8 encoded, build tools can have
bugs, and you don't want want the install to fail just because the log
isn't correctly encoded.

Hence, I think a 'SHOULD' is appropriate for this part of the spec:

- To install tool authors, it is clear that they can display the output
as UTF-8 so long as they don't crash if it's invalid.
- To build tool authors, it's clear that they can't pass the buck to
install tool authors if output gets jumbled because it's not UTF-8.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

I have made a PR against the PEP with my best take on the encoding
situation:
https://github.com/python/peps/pull/264/files

On Mon, May 22, 2017, at 11:19 AM, Paul Moore wrote:
> On 22 May 2017 at 10:56, Thomas Kluyver <tho...@kluyver.me.uk> wrote:
> > On Sat, May 20, 2017, at 07:36 PM, Steve Dower wrote:
> >> Require that build tools either send UTF-8 to the UI component, or write
> >> bytes to a file and call it a build output. I see no benefit in
> >> requiring both the build tool and the UI tool to guess what the text
> >> encoding is.
> >
> > I'm not proposing that the install tool should try to guess the
> > encoding, but I think a well written install tool shouldn't crash if the
> > build output doesn't match the encoding it expects. Even if the spec
> > says that the build output MUST be UTF-8 encoded, build tools can have
> > bugs, and you don't want want the install to fail just because the log
> > isn't correctly encoded.
> >
> > Hence, I think a 'SHOULD' is appropriate for this part of the spec:
> >
> > - To install tool authors, it is clear that they can display the output
> > as UTF-8 so long as they don't crash if it's invalid.
> > - To build tool authors, it's clear that they can't pass the buck to
> > install tool authors if output gets jumbled because it's not UTF-8.
> 
> I'd say that it's not so much just "well written" install tools. I'd
> say that install tools MUST NOT crash if build tool output isn't in
> the expected encoding. On the other hand, the encoding agreement
> implies that if build tools *do* send data in the correct encoding
> then they are entitled to expect that it will be displayed accurately
> to the end user.
> 
> Output can be garbled in two ways:
> 
> 1. The build tool does not (or cannot) ensure that its output is in
> the standard-mandated encoding.
> 2. The install tool cannot display the full range of characters
> representable in the standard-mandated encoding.
> 
> Neither of these should cause a failure. Well written install tools
> should warn in the case of (1) - "I have been passed data that I don't
> understand, I'll do my best to display it but can't guarantee the
> output won't be garbled". In the case of (2), though, that's "as
> expected" - if your OS settings mean you can't display certain
> characters, you shouldn't be surprised if your install tool replaces
> them with a placeholder.
> 
> On an implementation note, this boils down to something like the
> following in the install tool:
> 
> # Step 1
> try:
> data = decode build output using STD_ENCODING
> except UnicodeDecodeError:
> warn "Data is not in expected encoding"
> data = decode using STD_ENCODING with errors= replacement>
> 
> # Step 2
> data = data.encode(MY_OUTPUT_ENCODING, errors= replacement>).decode(MY_OUTPUT_ENCODING)
> 
> # We now have subprocess output that's safe to display if requested.
> 
> As a side note, I find step 2 "sanitise my string to ensure it can be
> safely output" to be a pretty common operation - possibly because
> Python's standard IO streams raise exceptions on unicode errors - and
> I'm surprised there isn't a better way to spell it than the
> encode/decode pair above.
> 
> Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

On Mon, May 22, 2017, at 12:02 PM, Paul Moore wrote:
> The only reservation I have is that the choice of UTF-8 means that on
> Windows, build backends pretty much have to explicitly manage tool
> output (as they are pretty much certain *not* to output in UTF-8).
> Build backend writers that aren't aware of this issue (most likely
> because their main platform is not Windows) could very easily choose
> to just pass through the raw bytes, and as a result *all* non-ASCII
> output would be garbled on non-UTF-8 systems.
> 
> Would locale.getpreferredencoding() not be a better choice here? I
> know it has issues in some situations on Unix, but are they worse than
> the issues UTF-8 would cause on Windows? After all it's the encoding
> used by subprocess.Popen in "universal newlines" mode...

What if it wants to send a character which can't be encoded in the
locale encoding? It's quite easy on Windows to end up with a character
that you can't encode as cp1252. If the build tool uses .encode(loc_enc,
'replace'), then you've lost information even before it gets to the
install tool.

It's 2017, I really don't want to go down the 'locale specified
encoding' route again. UTF-8 everywhere!

One affordance I'd consider is a recommendation to install tools that if
captured output is not valid UTF-8, they dump the raw bytes to a file so
that no information is lost. I'm not sure if that recommendation needs
to be in the spec itself, though.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

On Thu, May 25, 2017, at 03:38 PM, Nick Coghlan wrote:
> Seeing it like this pushes me from "Eh, maybe?" to "No, definitely not"
> [on the log directory] :)

That's fine by me. It does feel like unwanted extra complexity for both
backends and frontends. And backends dealing with output in an unknown
encoding can still choose to write it to a file and log "full output in
/tmp/blah" if they want - they don't need a spec for that.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

On Thu, May 25, 2017, at 02:27 PM, Paul Moore wrote:
> I'd be concerned here that we risk making the frontend UI a lot more
> complex for little actual benefit. I'd rather we stick with the
> current model, where a backend just has some output to pass through to
> the frontend. Let's get a solution that works for that before adding
> extra complexity, or we'll never get the PEP signed off.

I'm inclined to agree that we're overcomplicating things. But if we
can't agree on which simple-but-imperfect option to take, maybe it's
worth trying to work out something more complex.

My proposed addition to the PEP so far says this:

The build frontend may capture stdout and/or stderr from the backend. If
the backend detects that an output stream is not a terminal/console
(e.g. ``not sys.stdout.isatty()``), it SHOULD ensure that any output it
writes to that stream is UTF-8 encoded. The build frontend MUST NOT fail
if captured output is not valid UTF-8, but it MAY not preserve all the
information in that case (e.g. it may decode using the *replace* error
handler in Python). If the output stream is a terminal, the build
backend is responsible for presenting its output accurately, as for any
program running in a terminal.

We could add a paragraph like this:

The backend may do some operations, such as running subprocesses, which
produce output in an unknown encoding. To handle such output, the build
frontend MAY (?) create an empty directory, and set the environment
variable PEP517_BUILD_LOGS to the path of this directory for the
backend. If this environment variable is set, the backend MAY create any
number of files inside this directory containing additional output. This
is designed to allow the use of encoding detection tools on this output.
If files are created in this directory, frontends SHOULD display its
location in their output, and MAY display the contents of the files.

That's not a massive amount more complexity for the spec, but it does
add a moderate burden to frontend & backend implementations which want
to properly support it.

If you're being purist about it, displaying a path on a Unix based
system is producing output in an unknown encoding, since filenames in
Unix are bytes. I don't imagine many tools are going to go that far,
though.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Thomas Kluyver]

On Fri, May 19, 2017, at 03:41 PM, Paul Moore wrote:
> Can we specify what encoding the informational text must be written
> in?

Sure, that makes sense. What about:

All hooks are run with working directory set to the root of the source
tree, and MAY print arbitrary informational text on stdout and stderr.
This text SHOULD be UTF-8 encoded, but as building may invoke other
processes, install tools MUST NOT fail if the data they receive is not
valid UTF-8; though in this case the display of the output may be
corrupted.

Do we also want to recommend that install tools set
PYTHONIOENCODING=utf-8 when invoking build tools? Or leave this up to
the build tools?

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig