On 20May2017 1315, Paul Moore wrote:
On 20 May 2017 at 17:36, Steve Dower <steve.do...@python.org> wrote:
In general, since most subprocesses (at least on Windows) do not have
customizable encodings, the tool that launches them needs to know what the
encoding is. Since we don't live in a Python 3.6 world quite yet, that means
the tool should read raw bytes from the compiler and encode them to UTF-8.
Did you spot my point that Visual C produces output that's a mixture
of OEM and ANSI codepages?
[SNIP]
Yes, and it's a perfect example of why the MSVC-specific wrapper should
be the one to deal with tool encodings. If you forward unencoded bytes
like this back to the UI, it will have to deal with the mixed encoding.
I'd be very surprised if build tool developers got this sort of edge
case correct without at least some guidance from the PEP on the sorts
of things they need to consider. You suggest "read raw bytes and
encode them to UTF-8" - but you don't encode bytes, you encode
strings, so you still need to convert those bytes to a string first,
and there's no encoding you can reliably use for this. You need to use
"errors=replace" to ensure you can handle inconsistently encoded data
without getting an exception.
The "read raw bytes and [transcode] them" comment was meant to be that
sort of help. I didn't go as far as writing
`output.decode(output_encoding, errors="replace").encode("utf-8",
errors="replace")`, but that's basically what I meant to imply. The
build tool developer is the *only* developer who can get this right, and
if they can't, then they have to figure out the most appropriate way to
work around the fact that they can't.
As for defining distutils as incompatible with the PEP, I'm okay with
that. Updating distutils to use subprocess for launching tools rather
than spawnv would be a very good start (and likely a good contribution
for a new contributor), but allowing build tools to continue to be
written badly is not worthwhile.
Cheers,
Steve
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
