[issue8663] Failed compile in a non-ASCII path
Changes by STINNER Victor victor.stin...@haypocalc.com: Removed file: http://bugs.python.org/file17342/distutils_spawn_log.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: Although, all these patchs should have some tests demonstrating the bugs As discussed on IRC, fixing distutils.log package instead of setup.py is better. New patch includes unit tests. -- Added file: http://bugs.python.org/file17401/distutils_log_backslashreplace-2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: I commited to patch on distutils.log in r81359 (py3k). I'm waiting for the buildbot before backporting to 3.1. -- resolution: - fixed status: open - pending ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: I commited to patch on distutils.log in r81359 (py3k) Backported to 3.1 as r81363. -- status: pending - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
Changes by STINNER Victor victor.stin...@haypocalc.com: -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
Tarek Ziadé ziade.ta...@gmail.com added the comment: I thought you had a unit test, I don't see any in your commit -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: I thought you had a unit test, I don't see any in your commit patch -p0 ... svn ci doesn't include new files. I forgot it. r81361 includes the new file. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: New patch: use sys.stdout.encoding instead of ASCII. (If stdout is not a TTY, sys.stdout.encoding is ASCII.) -- Added file: http://bugs.python.org/file17342/distutils_spawn_log.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
Changes by STINNER Victor victor.stin...@haypocalc.com: Removed file: http://bugs.python.org/file17268/distutils_spawn_toascii.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: Conditions to reproduce the bug: - non ASCII directory - don't use make -j N (no MAKEFLAGS environment variable) - write make output into a pipe, eg. make 21|cat My 2 last patches are not enough: there are other functions writing non-ASCII strings to log whereas log encoding is ASCII. The right fix is to use the backslashreplace error handler for the log. Two solutions: a) replace sys.stdout by a new file using backslashreplace: I tried this solution for regrtest.py: #8533. My patch for regrtest.py doesn't work on Windows because of a newline issue b) emulate backslashreplace only in distutils log I prefer (a) because it can be implemented in setup.py without touching distutils (tarek told me that distutils shouldn't be patched too much). distutils_log_backslashreplace.patch implements (b). -- Added file: http://bugs.python.org/file17346/distutils_log_backslashreplace.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
Tarek Ziadé ziade.ta...@gmail.com added the comment: I'll look at that asap. Although, all these patchs should have some tests demonstrating the bugs -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: setup_stdout_backslashreplace.patch implements point (a). I don't know which solution is better. This issue is not specific to compiling CPython, other programs may fail in non-ASCII paths and so solution (b) is maybe better. -- Added file: http://bugs.python.org/file17347/setup_stdout_backslashreplace.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
New submission from Antoine Pitrou pit...@free.fr:
Distutils fails logging characters which are not part of the default encoding:
http://www.python.org/dev/buildbot/builders/AMD64%20Ubuntu%20wide%203.x/builds/1062/steps/compile/logs/stdio
[...]
File
/home/buildbot/cpython-ucs4-nonascii-\u20ac/3.x.pitrou-ubuntu-wide/build/Lib/distutils/dir_util.py,
line 67, in mkpath
log.info(creating %s, head)
File
/home/buildbot/cpython-ucs4-nonascii-\u20ac/3.x.pitrou-ubuntu-wide/build/Lib/distutils/log.py,
line 40, in info
self._log(INFO, msg, args)
File
/home/buildbot/cpython-ucs4-nonascii-\u20ac/3.x.pitrou-ubuntu-wide/build/Lib/distutils/log.py,
line 30, in _log
stream.write('%s\n' % msg)
UnicodeEncodeError: 'ascii' codec can't encode character '\u20ac' in position
81: ordinal not in range(128)
[54439 refs]
--
assignee: tarek
components: Distutils
messages: 105320
nosy: haypo, pitrou, tarek
priority: normal
severity: normal
stage: needs patch
status: open
title: Failed compile in a non-ASCII path
type: compile error
versions: Python 3.1, Python 3.2
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8663
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: What is your locale and your locale encoding? distutils use ASCII but I'm not sure that your locale encoding is ASCII, because Python fails to start if the locale is ASCII and the path contains a non ASCII character. See also issue #8611: Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
Antoine Pitrou pit...@free.fr added the comment: As I answered on IRC, this is on a buildbot environment. Compiling from the command line works fine, but not from the buildbot environment. From the command line, the locale is as following: $ locale LANG=en_GB.UTF-8 LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=en_GB.UTF-8 LC_TIME=en_GB.UTF-8 LC_COLLATE=en_GB.UTF-8 LC_MONETARY=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8 LC_PAPER=en_GB.UTF-8 LC_NAME=en_GB.UTF-8 LC_ADDRESS=en_GB.UTF-8 LC_TELEPHONE=en_GB.UTF-8 LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=en_GB.UTF-8 LC_ALL= But I guess this information is not useful. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
Antoine Pitrou pit...@free.fr added the comment: Ok, this might be more interesting: $ cat | ./python -i Python 3.2a0 (py3k:81010, May 8 2010, 23:25:47) [GCC 4.3.2] on linux2 Type help, copyright, credits or license for more information. import locale locale.getpreferredencoding() 'UTF-8' import sys sys.stdin _io.TextIOWrapper name='stdin' encoding='ascii' sys.getdefaultencoding() 'utf-8' As you see, stdin uses ascii... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
Antoine Pitrou pit...@free.fr added the comment: Anyway, regardless of the actual stdout encoding, distutils should be able to log messages without crashing, IMO. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: Conditions to reproduce the bug: don't use make -j N (unset MAKEFLAGS), stdout and stderr should be be TTY = use make 21|cat. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
Éric Araujo mer...@netwok.org added the comment: Another long-standing encoding bug was fixed: It was impossible to use e.g. an author name with non-ASCII characters. The fix makes distutils use UTF-8. Of course, it’s more complicated to choose an encoding for terminal I/O. -- nosy: +merwok ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
Éric Araujo mer...@netwok.org added the comment: (I wrote before I saw Victor’s reply) Does it work with PYTHONIOENDODING set to UTF-8? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: If the standard output is not a TTY, Python uses ASCII encoding for sys.stdout: ./python -c import sys;print(sys.stdout.encoding)|cat = ascii. This issue remembers me: #8533 (regrtest: use backslashreplace error handler for stdout). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
Antoine Pitrou pit...@free.fr added the comment: If the standard output is not a TTY, Python uses ASCII encoding for sys.stdout We could perhaps fix this too, if python-dev agrees. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: Attached patch escapes non-ASCII characters of the log message using ASCII+backslashreplace (but keep unicode type). -- keywords: +patch Added file: http://bugs.python.org/file17268/distutils_spawn_toascii.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
Éric Araujo mer...@netwok.org added the comment: Wasn’t PYTHONIOENCODING added for such cases? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
Antoine Pitrou pit...@free.fr added the comment: Wasn’t PYTHONIOENCODING added for such cases? Yes, it was, but it's a very bad workaround. In most if not all cases, people will set PYTHONIOENCODING to their system's default encoding. Therefore, they shouldn't have to set an environment variable at all. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: If the standard output is not a TTY, Python uses ASCII encoding for sys.stdout We could perhaps fix this too, if python-dev agrees. Open a new issue please if you consider that as a bug. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8663] Failed compile in a non-ASCII path
STINNER Victor victor.stin...@haypocalc.com added the comment: I tried to recompile Python with export PYTHONIOENCODING=ascii but it doesn't fail. That's because the Makefile calls ./python -E ./setup.py -q build: -E ignores environment variables. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8663 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
