[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[STINNER Victor]

STINNER Victor added the comment:


New changeset 94a3694c3dda97e3bcb51264bf47d948c5424d84 by Victor Stinner in 
branch '2.7':
bpo-6393: Fix locale.getprerredencoding() on macOS (#1555)
https://github.com/python/cpython/commit/94a3694c3dda97e3bcb51264bf47d948c5424d84


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +1651

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Sridhar Ratnakumar]

Changes by Sridhar Ratnakumar sridh...@activestate.com:


--
nosy: +srid

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Ronald Oussoren]

Changes by Ronald Oussoren ronaldousso...@mac.com:


--
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Ned Deily]

Ned Deily n...@acm.org added the comment:

The missing return result in the else case has been subsequently fixed in 
r75539 (py3k) and r75541 (3.0) so this issue should be re-closed.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Ronald Oussoren]

Ronald Oussoren ronaldousso...@mac.com added the comment:

I've applied the fixed version of my patch in r74687 (3.x) and r74688 
(3.1).

--
resolution:  - fixed
stage:  - committed/rejected
status: open - pending

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Graham Dumpleton]

Graham Dumpleton graham.dumple...@gmail.com added the comment:

I see this problem on both MacOS X 10.5 and on Windows. This is when using 
Python embedded inside of Apache/mod_wsgi.

On MacOS X the error is:

Fatal Python error: Py_Initialize: can't initialize sys standard streams
ImportError: No module named encodings.utf_8

On Windows the error is:

Fatal Python error: Py_Initialize: can't initialize sys standard streams
LookupError: unknown encoding: cp0

The talk about the fix mentioned it only addressing MacOS X. What about 
Windows case I am seeing. Will it help with that at all?

--
nosy: +grahamd

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Graham Dumpleton]

Graham Dumpleton graham.dumple...@gmail.com added the comment:

Hmmm, actually my MacOS X error is different, although Windows one is 
same, except that encoding is listed and isn't empty.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Graham Dumpleton]

Graham Dumpleton graham.dumple...@gmail.com added the comment:

You can ignore my MacOS X example as that was caused by something else.

My question still stands as to whether the fix will address the similar 
problem I saw on Windows.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Graham Dumpleton]

Graham Dumpleton graham.dumple...@gmail.com added the comment:

I have created issue6501 for my Windows variant of this problem given that 
it appears to be subtly different due to there being an encoding where as 
the MacOS X variant doesn't have one.

Seeing that the fix for the MacOS X issue is in Python code, I will when I 
have a chance look at whether can work out any fix for the Windows 
variant. Not sure I have right tools to compile Python from C code on 
Windows, so if a C code problem, not sure can really investigate.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Once this patch is checked in, should we do an emergency 3.1.1 release?

--
nosy: +benjamin.peterson, pitrou
priority:  - critical
versions: +Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

I'm don't know whether this is really worth a 3.1.1, all by itself.  
There's an easy workaround, which is for affected users to set their 
locale properly.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

 One *could* add a check in pythonrun.c to substitute some suitable
 default (UTF-8) if nl_langinfo(CODESET) returns an empty value.

While googling for the source of this problem, I found other software 
projects that take this approach.  It doesn't seem totally unreasonable.

I just wish I understood *why* nl_langinfo(CODESET) is returning  in 
these cases.  I've looked for the source at 
http://www.opensource.apple.com, but can't find it;  maybe that part of 
Darwin isn't open source.

It seems that a lot of people end up with an OS X Terminal setup such that 
LC_CTYPE is 'UTF-8' (perhaps this is a 10.4 thing---I haven't encountered 
this myself);  I don't think these people should have to deal with a 
confusing error on startup;  defaulting to UTF-8 on OS X seems like a 
reasonable compromise.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Ronald Oussoren]

Ronald Oussoren ronaldousso...@mac.com added the comment:

The manpage says that nl_langinfo returns an empty string when there is 
an invalid setting.

There is validity in saying that 'LANG=utf-8' is an invalid setting, the 
LANG variable is supposed to a locale name, which would be a language 
setting (possibly combined with a codeset definition). utf-8 is not a 
language.

I wouldn't mind falling back to utf-8 as the default codeset when 
nl_langinfo returns an empty string because utf-8 is the default 
character set on OSX, and furthermore defaulting to some value is way 
better than crashing.

I do wonder how the user ended up with LANG=utf-8 in the first place.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

 There is validity in saying that 'LANG=utf-8' is an invalid setting

Agreed.  But that doesn't really explain why e.g. LANG=en_US also 
produces , while LANG=invalid produces US-ASCII.

 I do wonder how the user ended up with LANG=utf-8 in the first place.

Me too.  As far as I can gather, it's a result of setting the Terminal 
preferences (particularly the character encoding and 'Set LANG 
environment variable on startup' checkbox) in some particular way, on 
some versions of OS X, for users in some countries, at some particular 
phases of the moon, etc...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Ronald Oussoren]

Ronald Oussoren ronaldousso...@mac.com added the comment:

The attached patch (issue6393-fix.patch) seems to fix the issue.

Could you please test and have a look at the patch? It basicly tests if 
the output of nl_langinfo(CODESET) is the empty string and defaults to 
'UTF-8' in that case (but only on OSX).

I intent to apply this patch unless someone objects to that.

--
keywords: +needs review, patch
Added file: http://bugs.python.org/file14476/issue6393-fix.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

Thanks, Ronald!  The patch fixes the problem for me.
(I directly patched the locale.py file installed from
the Python dmg, since I still haven't figured out how
to build a python executable that exhibits this
problem.)

The patch doesn't look quite right, though: in the else clause,
it looks as though you're testing 'result' before it exists.
Shouldn't the 'result = nl_langinfo(CODESET)' line come
before the 'if not result and ' line?

On the subject of Terminal and LANG, LC_CTYPE settings, I found an 
interesting link:

http://pastie.textmate.org/111807

Indeed, after setting my region to 'South Africa' in Preferences - 
International - Formats, a newly opened Terminal window gives me:

newton:~ dickinsm$ locale
LANG=
LC_COLLATE=C
LC_CTYPE=UTF-8
LC_MESSAGES=C
LC_MONETARY=C
LC_NUMERIC=C
LC_TIME=C
LC_ALL=

And then python3 crashes on startup as above.  This is on a newborn (3-
week old) MacBook Pro that's been barely changed from default settings 
(and no transfer of files and settings from an old Mac, either).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Ronald Oussoren]

Ronald Oussoren ronaldousso...@mac.com added the comment:

Good catch, the code in the else is indeed in the wrong order.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Ned Deily]

Ned Deily n...@acm.org added the comment:

Looks good and the patched patch also works in a py3k installer build.

BTW, Mark, I was curious as to why you were unable to reproduce the 
problem with your own build.  I should have mentioned that my testing 
was with complete installer (framework) builds.  I subsequently 
experimented with a non-framework build and found that I could not 
reproduce the problem running from the ./python in the build directory.  
Stepping through gdb showed that, during the calls from create_stdio, 
the import of locale fails in textio.c, so it falls back to using 
ascii as the default encoding (~line 899) and avoids the crash.  If I 
do a make install, the unpatched installed bin/python3 does crash in the 
same way as with the installer python3.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Philipp]

Changes by Philipp pkal...@googlemail.com:


--
nosy: +Phil

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Ned Deily]

Ned Deily n...@acm.org added the comment:

This is a side effect of the fix for Issue6202.  Prior to r73268, 
locale.getpreferredencoding always returned mac-roman regardless of the 
setting of LANG, so this wasn't a problem in py3k (or 3.0.x builds) up 
through 3.1rc1.  I can reproduce it on current py3k and release31-maint.

--
nosy: +nad

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Ned Deily]

Ned Deily n...@acm.org added the comment:

Note, you can produce the same error on OS X or linux by setting 
PYTHONIOENCODING=, which effectively overrides the value returned 
nl_langinfo(CODESET).  In pythonrun.c, create_stdio passes 
PYTHONENCODING, if set, on as the encoding value to TextIOWrapper.  If 
no encoding was specified, TextIOWrapper uses the value returned by 
locale.getpreferrencoding().  It then calls PyCodec_IncrementalDecoder 
and the unknown (or empty) encoding is finally detected.

That raises the question of how far python should go in protecting the 
user.  One *could* add a check in pythonrun.c to substitute some 
suitable default (UTF-8) if nl_langinfo(CODESET) returns an empty value.  
Or perhaps just abort there with a more meaningful error message.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Ned Deily]

Ned Deily n...@acm.org added the comment:

... create_stdio passes PYTHONIOENCODING ...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6393] OS X: python3 from python-3.1.dmg crashes at startup

[Mark Dickinson]

New submission from Mark Dickinson dicki...@gmail.com:

There was a report[1] on c.l.p. that python3 from the OS X Python 3.1 
dmg download at www.python.org/download/releases/3.1/ crashes on 
startup.  I can reproduce this with the python.org download (using the 
OS X Terminal) only with a bad locale setting:

newton:~ dickinsm$ LANG=utf-8 python3
Fatal Python error: Py_Initialize: can't initialize sys standard streams
LookupError: unknown encoding: 
Abort trap (core dumped)

The core dump isn't useful:  just lots of 'No symbol table info 
available.'

This is on OS X 10.5.7/Intel.

I can't reproduce it with either the py3k branch or the release31-maint 
branch, built from scratch.

I suspect that this has to do with the behaviour of nl_langinfo(CODESET) 
on OS X: namely, after doing (in C) setlocale(LC_CTYPE, ), the result 
of nl_langinfo(CODESET) appears to be UTF-8 for well-defined utf-8 
locales (e.g., 'en_US.UTF-8'), US-ASCII for meaningless locales (e.g., 
'invalid'), but one just gets  for locales like 'utf-8' or 'en_US'.
This in turn affects Python's locale.getpreferredencoding function. 
See also issue 2173, which may be related.

Ronald, any ideas?

[1] http://mail.python.org/pipermail/python-list/2009-June/718255.html

--
assignee: ronaldoussoren
components: Interpreter Core, Macintosh
messages: 89972
nosy: marketdickinson, ronaldoussoren
severity: normal
status: open
title: OS X: python3 from python-3.1.dmg crashes at startup
type: crash
versions: Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6393
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com