[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Martin v. Löwis mar...@v.loewis.de added the comment: It looks like a bug fix to me - so it would apply to all four active branches. -- status: pending - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Jeroen Ruigrok van der Werven asmo...@in-nomine.org added the comment: Committed in r72381 and r72395. -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Jeroen Ruigrok van der Werven asmo...@in-nomine.org added the comment: Committed the initial patch in r72375 for trunk and r72376 for py3k. Any other branches that would need the merge? 3.0? -- resolution: - accepted stage: test needed - committed/rejected status: open - pending ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Georg Brandl ge...@python.org added the comment: Interestingly, my setlocale(3p) man page says: ERRORS No errors are defined. So isn't it debatable if returning the NULL pointer really is an error? -- nosy: +georg.brandl ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Jeroen Ruigrok van der Werven asmo...@in-nomine.org added the comment: On the subject whether or not returning a null pointer should be considered he said: - On the subject whether or not returning a null pointer should be considered an error he said: -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Changes by Jeroen Ruigrok van der Werven asmo...@in-nomine.org: Removed file: http://bugs.python.org/file13843/locale.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Changes by Jeroen Ruigrok van der Werven asmo...@in-nomine.org: Removed file: http://bugs.python.org/file13849/locale.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Jeroen Ruigrok van der Werven asmo...@in-nomine.org added the comment: Really correct this time. -- Added file: http://bugs.python.org/file13850/locale.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Jeroen Ruigrok van der Werven asmo...@in-nomine.org added the comment: Georg pointed out a mistake I introduced in my patch, updated now. -- Added file: http://bugs.python.org/file13849/locale.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Jeroen Ruigrok van der Werven asmo...@in-nomine.org added the comment: I asked that as well on the POSIX/SUS list and Don Cragun responded with: If you make the last argument to setlocale() be a pointer to unallocated memory, implementations would be allowed to set errno to EFAULT and terminate the process with a core dump even when this section says No errors are defined. An implementation could also set errno to ENOENT (e.g., if the B locale wasn't known) or to EINVAL (e.g., if the B locale existed but the LC_CTYPE portion of the locale was not in the proper format). That wording just means that the standard doesn't require implementations to detect errors like these nor to report specific error values for different possible errors. On the subject whether or not returning a null pointer should be considered he said: The standard is silent on this issue. Why does it make any difference to an application? If setlocale(LC_CTYPE, B) returns a null pointer, the LC_CTYPE portion of the locale was not changed. If setlocale(LC_CTYPE, B) does not return a null pointer, the LC_CTYPE portion of the locale was successfully changed. I am just wondering why we want to be quite different from how many other languages are approaching the issue. Sure enough, we can use a try: construct, but it kind of defeats the principle of least astonishment by being different from the rest on this issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Martin v. Löwis mar...@v.loewis.de added the comment: ERRORS No errors are defined. So isn't it debatable if returning the NULL pointer really is an error? As Jeroen reports, this really means two different things a) no errors really means no errno codes. Whether or not an error may occur is an independent issue. b) are defined really means that POSIX doesn't define any standard errno codes; the system may indeed still set errno (C99, 7.5p3) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Martin v. Löwis mar...@v.loewis.de added the comment: I am just wondering why we want to be quite different from how many other languages are approaching the issue. Because we have exceptions, and they don't. Would you also propose that open() should return None, just because fopen(3) returns NULL? While it may be debatable whether applications care about the error when passing as the locale, there is also the second case where applications pass an explicit locale setlocale(locale.LC_ALL, de...@euro) When they do that, they surely want to be told if this actually worked. Sure enough, we can use a try: construct, but it kind of defeats the principle of least astonishment by being different from the rest on this issue. There is also the backwards compatibility issue: your change will break existing code. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
R. David Murray rdmur...@bitdance.com added the comment: On Sun, 3 May 2009 at 08:55, Jeroen Ruigrok van der Werven wrote: I am just wondering why we want to be quite different from how many other languages are approaching the issue. Sure enough, we can use a try: construct, but it kind of defeats the principle of least astonishment by being different from the rest on this issue. Only if you imagine that the principal applies to expectations inherited from other languages. In a Python context, which is what the principle actually refers to, it would be astonishing if the error were to be silently ignored. -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Jeroen Ruigrok van der Werven asmo...@in-nomine.org added the comment:
I will first point out where our current implementation is broken, in my
opinion of course, after which I propose a small patch.
Both C90 (7.4.1.1) and C99 (7.11.1.1) state:
A value of C for locale specifies the minimal environment for C
translation; a value of for locale specifies the locale-specific
native environment. Other implementation-defined strings may be passed
as the second argument to setlocale.
[...]
If a pointer to a string is given for locale and the selection can be
honored, the setlocale function returns a pointer to the string
associated with the specified category for the new locale. If the
selection cannot be honored, the setlocale function returns a null
pointer and the program’s locale is not changed.
Note that neither C or POSIX defines any errors or sets errno or the
likes. It simply returns a null pointer.
In C you would typically start your program with a call like:
#include locale.h
int main(int argc, char *argv[]) {
setlocale(LC_CTYPE, );
...
}
This will try to set the locale to what the native environment
specifies, but will not error out if the value, if any, it receives does
not map to a valid locale. It will return a null pointer if it cannot
set the locale. Execution continues and the locale is set to the default
C.
Our current behaviour in Python does not adhere to these semantics. To
illustrate:
# Obvious non-existing locale
from locale import setlocale, LC_CTYPE
setlocale(LC_CTYPE, 'B')
Error: unsupported locale setting
# Valid locale, but not available on my system
from os import getenv
from locale import setlocale, LC_CTYPE
getenv('LANG')
'cy_GB.UTF-8'
setlocale(LC_CTYPE, '')
Error: unsupported locale setting
Neither Perl or PHP throw any error when setlocale() is passed an
invalid locale. Python is being unnecessarily disruptive by throwing an
error.
As such I think PyLocale_setlocale() in Modules/_localemodule.c needs to
be adjusted. Patch against trunk enclosed. This changes the semantics of
our current implementation to the following:
from locale import setlocale, LC_CTYPE
rv = setlocale(LC_CTYPE, 'B')
type(rv)
class 'NoneType'
rv = setlocale(LC_CTYPE, 'C')
type(rv)
class 'str'
rv
'C'
--
Added file: http://bugs.python.org/file13843/locale.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1443504
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Martin v. Löwis mar...@v.loewis.de added the comment:
If a pointer to a string is given for locale and the selection can be
honored, the setlocale function returns a pointer to the string
associated with the specified category for the new locale. If the
selection cannot be honored, the setlocale function returns a null
pointer and the program’s locale is not changed.
Note that neither C or POSIX defines any errors or sets errno or the
likes. It simply returns a null pointer.
Still, this is considered as an error case.
#include locale.h
int main(int argc, char *argv[]) {
setlocale(LC_CTYPE, );
...
}
This will try to set the locale to what the native environment
specifies, but will not error out if the value
Yes, but that's a bug in the C code, which fails to check the
return value of setlocale. The fact that the bug is wide-spread
doesn't make it any better.
As such I think PyLocale_setlocale() in Modules/_localemodule.c needs to
be adjusted
-1. Errors should never pass silently. That's the whole point of exceptions.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1443504
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Martin v. Löwis mar...@v.loewis.de added the comment: The patch looks fine to me. -- nosy: +loewis ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Martin v. Löwis mar...@v.loewis.de added the comment: But I am curious about your thoughts about the _parse_localename() method being called from setlocale() raising a ValueError, whereas a setlocale(LC_CTYPE, ) should not fail at all, which it currently does if the locale in the environment is not valid. I fail to see how this is related to this issue. In the OP's report, the exception was locale.Error, not ValueError, and _parse_localename isn't ever being called from setlocale() - why do you think it is being called? AFAICT, the only callers of _parse_localename are getlocale and getdefaultlocale (which, IMO, should both be deprecated). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Jeroen Ruigrok van der Werven asmo...@in-nomine.org added the comment: Sorry, I was actually off by a method last night. It turns out the problem lies in _localemodule.c. Let me start with the basic question: is our setlocale() supposed to mirror POSIX' operations/semantics? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Martin v. Löwis mar...@v.loewis.de added the comment: Let me start with the basic question: is our setlocale() supposed to mirror POSIX' operations/semantics? Yes, it is. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Jeroen Ruigrok van der Werven asmo...@in-nomine.org added the comment: Shouldn't the fallback be to setlocale(LC_CTYPE, C) instead of silently passing, though? -- nosy: +asmodai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Jeroen Ruigrok van der Werven asmo...@in-nomine.org added the comment: You don't want to completely nix the setlocale(LC_CTYPE, ) call though. The denotes to grab the native environment, in other words, to grab whatever the current user's LC_CTYPE environment variable is set to (see `locale -a`) and then set the program's LC_CTYPE to that. Of course, this might be set to something that might be valid (e.g. cy_GB.ISO8859-15), but has no matching entry in /usr/share/locale (or wherever your system provides it) and as such it fails. Reading SUS (The Single Unix Specification) I see that it explicitly says: Upon successful completion, setlocale() shall return the string associated with the specified category for the new locale. Otherwise, setlocale() shall return a null pointer and the locale of the process is not changed. So the patch seems to be correct actually. We try to setlocale(LC_CTYPE, ) to grab a locale from the environment to set LC_CTYPE, but we fail for whatever, so we should just pass since we should not adjust LC_CTYPE. Mmm, but it seems setlocale() in locale.py is not adhering to the standard by not allowing the case properly. _parse_localename() is being overly pedantic about this by raising a ValueError. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Changes by Daniel Diniz aja...@gmail.com: -- keywords: +patch stage: - test needed type: - behavior versions: +Python 2.6, Python 3.0 -Python 2.4, Python 2.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1443504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1443504] locale.getpreferredencoding() dies when setlocale fails
Heikki Toivonen added the comment: We noticed this too in Chandler. We worked around this issue with the patch I am attaching. Maybe not a correct fix, though. -- nosy: +heikki versions: +Python 2.5 Added file: http://bugs.python.org/file8833/patches-2.5.1-Linux _ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1443504 _ patches-2.5.1-Linux Description: Binary data ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
