[issue24968] Python 3 raises Unicode errors with the xxx.UTF-8 locale
Roberto Sánchez added the comment: Ok, that makes sense, besides David pointed me about another opened issue that could help to solve cases like this: http://bugs.python.org/issue15216 If the encoding is wrong because the environment but we can change the initial stream encodings (in stdin/out) easily we have a powerful tool to adapt our scripts and patch broken locales like the generated with SSH sessions. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24968> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24966] shutil.get_terminal_size() throws ValueError if stdout is detached, no fallback
Roberto Sánchez added the comment: Yeah, sorry, actually I was thinking in the stdin/out section, the detach() doc itself is Ok, the problem comes when is called in stdin/out. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24966> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24966] shutil.get_terminal_size() throws ValueError if stdout is detached, no fallback
Roberto Sánchez added the comment: In my case, I modify the stdout but the error was thrown by a third party (click lib), I need to overwrite the stdout because the program has a CLI and the locale env in the terminal seems to be wrong: codecs.lookup(locale.getpreferredencoding()).name == 'ascii' sys.getdefaultencoding() == 'utf-8' For these cases I try to fix it forcing the the IO encoding to the value given by sys.getdefaultencoding(), otherwise the script in a CLI won't work properly. I can apply a monkey-patch on the third party method that return "os.get_terminal_size()" if a ValueError exception is thrown, actually, The script is working now with the monkey-patch, but I thought that a bit of more "magic" in get_terminal_size would help to deal with the encoding issues in python3 CLI programs. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24966> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24966] shutil.get_terminal_size() throws ValueError if stdout is detached, no fallback
Roberto Sánchez added the comment: Thank you for the references, changing the encoding of an open stream sounds interesting, but It's targeted for v. 3.6 so It seems a bit far away. Anyway, as I said I fixed on my side with monkey-patching on the third-party lib, the broken locale is just the context, just to explain how I got this issue, I opened this because I honestly though that could be better to return the fallback value that raise an exception, but you know better the "big picture" so, if It's better as is, It's ok for me. BTW, I don't find the opened issue to use utf8 as default instead ascii (except for Windows) -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24966> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24966] shutil.get_terminal_size() throws ValueError if stdout is detached, no fallback
Roberto Sánchez added the comment: I totally agree, on the other hand, if the detach() method can cause that some core methods like shutil.get_termina_size() raise an exception then I think that should be warned in the doc of detach(). In all places that I've seen, the detach() method is used to overwrite the stdout in py3, so a sort of advice about the using of detach() It'd save some problems. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24966> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24968] Python 3 raises Unicode errors with the xxx.UTF-8 locale
New submission from Roberto Sánchez:
System: Python 3.4.2 on Linux Fedora 22
This issues is strongly related with: http://bugs.python.org/issue19846 But It
isn't exactly the same case.
When I connect from my Mac OSX (using Terminal.app) to a Linux host with Fedora
through ssh, the terminal session is forced to the OSX locale (default behavior
in Terminal.app):
[rob@fedora22 ~]$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=es_ES.UTF-8
LC_CTYPE="es_ES.UTF-8"
LC_NUMERIC="es_ES.UTF-8"
LC_TIME="es_ES.UTF-8"
LC_COLLATE="es_ES.UTF-8"
LC_MONETARY="es_ES.UTF-8"
LC_MESSAGES="es_ES.UTF-8"
LC_PAPER="es_ES.UTF-8"
LC_NAME="es_ES.UTF-8"
LC_ADDRESS="es_ES.UTF-8"
LC_TELEPHONE="es_ES.UTF-8"
LC_MEASUREMENT="es_ES.UTF-8"
LC_IDENTIFICATION="es_ES.UTF-8"
LC_ALL=
However the installed locales in Fedora are:
[rob@fedora22 ~]$ localectl list-locales
en_US
en_US.iso88591
en_US.iso885915
en_US.utf8 <-- This is the default one
And if a launch python3 I get:
[rob@fedora22 ~]$ python3
Python 3.4.2 (default, Jul 9 2015, 17:24:30)
[GCC 5.1.1 20150618 (Red Hat 5.1.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, codecs, sys, locale
>>> locale.getpreferredencoding()
'ANSI_X3.4-1968'
>>> codecs.lookup(locale.getpreferredencoding()).name
'ascii'
>>> locale.getdefaultlocale()
('es_ES', 'UTF-8')
>>> sys.stdout.encoding
'ANSI_X3.4-1968'
>>> sys.getfilesystemencoding()
'ascii'
>>> print('España')
File "", line 0
^
SyntaxError: 'ascii' codec can't decode byte 0xc3 in position 11: ordinal
not in range(128)
So, If I'm understanding correctly, If the current locale is not supported by
the system then python fallback to ascii.
I can understand this behavior when the supported locales and the current one
has different encoding, but if both of them are 'utf-8' It sounds reasonable
that locale.getpreferredencoding() is set to 'utf-8'.
This case is causing that programs with CLI (Command Line Interface) fails, if
you are using a third party like click lib, a RuntimeException is thrown by the
own lib, I learned it by the hard way, the python3 CLI programs need a valid
encoding to deal with stdin/stdout, and in this case all systems seems
correctly configured about the encoding, I mean, this is a real case, there is
no manual locale config modification, IMHO the current behavior seems a bit
strict.
--
components: Unicode
messages: 249390
nosy: ezio.melotti, haypo, rsc1975
priority: normal
severity: normal
status: open
title: Python 3 raises Unicode errors with the xxx.UTF-8 locale
type: behavior
versions: Python 3.4
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue24968>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24966] shutil.get_terminal_size() throws ValueError if stdout is detached, no fallback
Roberto Sánchez added the comment: I've added a new issue: http://bugs.python.org/issue24968 that is related with this one, because is the root cause to modify the stdin/stdout streams. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24966> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24968] Python 3 raises Unicode errors with the xxx.UTF-8 locale
Roberto Sánchez added the comment: OK, I already knew that "It is not a bug", but the scenario seems quite common, connection to a Linux host from a Mac with Terminal.app and different locales (default behavior), so a bit of "magic" when the locale's encoding part is correct would help to deal with some Unicode issues in python3 scripts. I just say that It would be a desirable enhancement, but I have no idea how to complex can be to change the current behavior, maybe It isn't worth the effort. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24968> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24966] shutil.get_terminal_size() throws ValueError is stdout is detached, no fallback
New submission from Roberto Sánchez: After the stdout stream has been overwritten and detached, the method shutils.get_terminal_size throws a ValueError exception and no fallback value is returned. Code to reproduce it: import os, sys, codecs, shutils sys.stdout = codecs.getwriter(utf-8)(sys.stdout.detach()) os.get_terminal_size() os.terminal_size(columns=160, lines=37) shutil.get_terminal_size((200, 30)) Traceback (most recent call last): File stdin, line 1, in module File /usr/lib64/python3.4/shutil.py, line 1057, in get_terminal_size size = os.get_terminal_size(sys.__stdout__.fileno()) ValueError: underlying buffer has been detached Is this the expected behavior ? IMHO, the given fallback values should be returned. The official doc: If the terminal size cannot be successfully queried, either because the system doesn’t support querying, or because we are not connected to a terminal, the value given in fallback parameter is used. fallback defaults to (80, 24) which is the default size used by many terminal emulators BTW, the function os.get_terminal_size() returns the correct size when it's invoked without parameters, maybe It could be a good fallback if sys.__stdout__ is not available and there isn't any user fallback values. -- components: IO messages: 249372 nosy: rsc1975 priority: normal severity: normal status: open title: shutil.get_terminal_size() throws ValueError is stdout is detached, no fallback type: behavior versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24966 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24966] shutil.get_terminal_size() throws ValueError if stdout is detached, no fallback
Changes by Roberto Sánchez d...@r75.es: -- title: shutil.get_terminal_size() throws ValueError is stdout is detached, no fallback - shutil.get_terminal_size() throws ValueError if stdout is detached, no fallback ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24966 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
