[issue12010] Compile fails when sizeof(wchar_t) == 1

[Irit Katriel]

Irit Katriel  added the comment:

Closing after over 10 years of no activity. (The discussion from back then 
seems to be leaning towards wont fix.)

--
nosy: +iritkatriel
resolution:  -> out of date
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[Mark Lawrence]

Change by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[Mark Lawrence]

Mark Lawrence added the comment:

Is this effectively superseded by work being done on #23496?

--
nosy: +BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[David Coles]

David Coles coles.da...@gmail.com added the comment:

Attached is a patch that updates configure.in to make sure that wchar.h is 
present and that wchar_t is at least 16 bits wide.

On android-8 this patch causes the configure step to fail since SIZEOF_WCHAR_T 
== 1. On android-9 and my Linux desktop the build continues as normal, passing 
the build tests. If wchar.h is removed from the system then the build also 
fails with an error as expected.

The patch does not contain the configure diff since it also contained some 
other changes and I wasn't sure of the correct autoreconf process for Python.

--
keywords: +patch
Added file: http://bugs.python.org/file22012/wchar_check.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[David Coles]

David Coles coles.da...@gmail.com added the comment:

After doing some more investigation it appears that Android's wchar_t support 
before android-9 is totally broken (see 
http://android.git.kernel.org/?p=platform/ndk.git;a=blob_plain;f=docs/STANDALONE-TOOLCHAIN.html;hb=HEAD).
 With android-9 you get 4 byte wchar_t and working wide character functions.

Possibly of more interest for Python is that it's no longer buildable without 
wchar_t support. While unicodeobject is pretty good at checking HAVE_WCHAR_H, a 
number of modules and even pythonrun.c directly use wchar_t or functions like 
PyUnicode_FromWideChar without providing a fallback. Does Python 3 now require 
wchar_t or are these bugs? (either option seems sensible).

A few other notes:
HAVE_USABLE_WCHAR_T looks like it was a check for unsigned/16 bits wchar_t 
that would allow them to be directly memcpy'd. The code in unicodeobject.c 
seems not to really use this anymore except (it's happy with signed or 
unsigned) and it looks like the check is only used for Windows now.

To properly support wchar_t of size 1 you would basically implement multibyte 
character storage either with UTF-8 or just packing two wchar_t's with UTF-16. 
At least in Android the distinction doesn't seem to matter as Android's 
internationalziation/localization policy seems to be use Java.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 Possibly of more interest for Python is that it's no longer buildable
 without wchar_t support. While unicodeobject is pretty good at
 checking HAVE_WCHAR_H, a number of modules and even pythonrun.c
 directly use wchar_t or functions like PyUnicode_FromWideChar without
 providing a fallback. Does Python 3 now require wchar_t or are these
 bugs? (either option seems sensible).

It's pretty much required since we rely on mbstowcs and friends to
convert some 8-bit strings (such as environment variables, command-line
args...) to unicode.

 At least in Android the distinction doesn't seem to matter as
 Android's internationalziation/localization policy seems to be use
 Java.

Ha :-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[Marc-Andre Lemburg]

Marc-Andre Lemburg m...@egenix.com added the comment:

David Coles wrote:
 
 David Coles coles.da...@gmail.com added the comment:
 
 After doing some more investigation it appears that Android's wchar_t support 
 before android-9 is totally broken (see 
 http://android.git.kernel.org/?p=platform/ndk.git;a=blob_plain;f=docs/STANDALONE-TOOLCHAIN.html;hb=HEAD).
  With android-9 you get 4 byte wchar_t and working wide character functions.

 Possibly of more interest for Python is that it's no longer buildable without 
 wchar_t support. While unicodeobject is pretty good at checking HAVE_WCHAR_H, 
 a number of modules and even pythonrun.c directly use wchar_t or functions 
 like PyUnicode_FromWideChar without providing a fallback. Does Python 3 now 
 require wchar_t or are these bugs? (either option seems sensible).

wchar_t should be fairly portable these days. I think the main
problem is that we never assumed sizeof(wchar_t) == 1 to be a
possibility. On Windows, wchar_t was 16 bit and the glibc started
out with 32 bits.

 A few other notes:
 HAVE_USABLE_WCHAR_T looks like it was a check for unsigned/16 bits wchar_t 
 that would allow them to be directly memcpy'd. The code in unicodeobject.c 
 seems not to really use this anymore except (it's happy with signed or 
 unsigned) and it looks like the check is only used for Windows now.

Note that HAVE_USABLE_WCHAR_T is only used to check whether
Python can use wchar_t as alias for Py_UNICODE. Python's Unicode
implementation needs Py_UNICODE to be an unsigned type with
either 2 bytes or 4 bytes. If wchar_t does not provide these
sizes or is a signed type, Python cannot use it for Py_UNICODE
and must instead use unsigned short.

If the configure script does not detect this case, then a patch
would be helpful.

The other wchar_t C lib functions should still remain usable,
though.

 To properly support wchar_t of size 1 you would basically implement multibyte 
 character storage either with UTF-8 or just packing two wchar_t's with 
 UTF-16. At least in Android the distinction doesn't seem to matter as 
 Android's internationalziation/localization policy seems to be use Java.

Python should not use wchar_t for Py_UNICODE on such platforms
and instead go with unsigned short.

I would assume that the wchar_t C lib routines work based on UTF-8
with sizeof(wchar_t) == 1, so the PyUnicode_*WideChar*() APIs would
need to be adjusted to work more or less like the UTF-8 codecs.

--
nosy: +lemburg

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[Marc-Andre Lemburg]

Marc-Andre Lemburg m...@egenix.com added the comment:

From the document you posted:


As documented, the Android platform did not really support wchar_t until
Android 2.3. What this means in practical terms is that:

  - If you target platform android-9 or higher, the size of wchar_t is
4 bytes, and most wide-char functions are available in the C library
(with the exception of multi-byte encoding/decoding functions and
 wsprintf/wsscanf).

  - If you target any prior API level, the size of wchar_t will be 1 byte
and none of the wide-char functions will work anyway.

We recommend any developer to get rid of any dependencies on the wchar_t type
and switch to better representations. The support provided in Android is only
there to help you migrate existing code.


With none of the wide-char functions working in Android 2.3, I don't
think you have a good chance of getting Python 3.x working, unless
you remove all their uses in the code and replace them with standard
char* functions.

The last paragraph doesn't sound very promising either. I wonder
what they mean with better representation. The C standard doesn't
have any better representation for Unicode at the moment.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[Martin v . Löwis]

Martin v. Löwis mar...@v.loewis.de added the comment:

I think what they mean is a better representation from an Android API, such as 
UChar32 from utils/AndroidUnicode.h.

I agree that it's not worthwhile trying to port Python to those Android 
versions that have a single-byte wchar_t definition. 

David, I think you are misunderstanding the purpose of HAVE_USABLE_WCHAR_T: It 
does *not* specify whether wchar_t can be used. Instead, it specifies whether 
wchar_t can be used as the datatype for Py_UNICODE. Calling it 
HAVE_A_WCHAR_T_DEFINITION_THAT_IS_SUITABLE_FOR_USE_AS_BASE_TYPE_OF_PY_UNICODE 
was just a little too tedious :-)

--
nosy: +loewis

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[David Coles]

David Coles coles.da...@gmail.com added the comment:

On Fri, May 6, 2011 at 1:31 PM, Marc-Andre Lemburg
rep...@bugs.python.org wrote:
 wchar_t should be fairly portable these days. I think the main
 problem is that we never assumed sizeof(wchar_t) == 1 to be a
 possibility. On Windows, wchar_t was 16 bit and the glibc started
 out with 32 bits.

Well a 1 byte wchar_t is a bit ass backwards. I think it's very much
an edge case. :)

 Note that HAVE_USABLE_WCHAR_T is only used to check whether
 Python can use wchar_t as alias for Py_UNICODE. Python's Unicode
 implementation needs Py_UNICODE to be an unsigned type with
 either 2 bytes or 4 bytes. If wchar_t does not provide these
 sizes or is a signed type, Python cannot use it for Py_UNICODE
 and must instead use unsigned short.

Right. That makes sense. In that case it's probably sensible to keep around.

 If the configure script does not detect this case, then a patch
 would be helpful.

Yup. I'll put something together that causes configure to bail out if
you're either missing HAVE_WCHAR_H or if SIZEOF_WCHAR_T is less than
16 bits.

 Python should not use wchar_t for Py_UNICODE on such platforms
 and instead go with unsigned short.

 I would assume that the wchar_t C lib routines work based on UTF-8
 with sizeof(wchar_t) == 1, so the PyUnicode_*WideChar*() APIs would
 need to be adjusted to work more or less like the UTF-8 codecs.

Yes. Using UTF-8 would be the sensible solution. Sadly it looks like
all the wide character functions 2.3 are undefined, so in this case
Android saying it has wchar_t support is worse than useless.

On Fri, May 6, 2011 at 1:37 PM, Marc-Andre Lemburg
rep...@bugs.python.org wrote:
 With none of the wide-char functions working in Android 2.3, I don't
 think you have a good chance of getting Python 3.x working, unless
 you remove all their uses in the code and replace them with standard
 char* functions.

I agree. In my case I should be able to bump the required version
number without too much fuss. It seems a bit silly to write in support
for a platform that no longer supports said feature.

 The last paragraph doesn't sound very promising either. I wonder
 what they mean with better representation. The C standard doesn't
 have any better representation for Unicode at the moment.

In C I guess the only sensible alternative would be UTF-8 char strings
(or maybe using uint32_t), but in Python's case it really depends on
how the underlying OS represents internationalized characters. Perhaps
in other projects you would use an external library like ICU, but
that's out the scope of my experience. :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[David Coles]

David Coles coles.da...@gmail.com added the comment:

On Fri, May 6, 2011 at 2:24 PM, Martin v. Löwis rep...@bugs.python.org wrote:

 Martin v. Löwis mar...@v.loewis.de added the comment:

 I think what they mean is a better representation from an Android API, such 
 as UChar32 from utils/AndroidUnicode.h.

Ah. Sadly I don't think that's exposed in the NDK yet.

 I agree that it's not worthwhile trying to port Python to those Android 
 versions that have a single-byte wchar_t definition.

Yup. Will be using Android 2.3+. If I'm forced to use an earlier
version of Android I think it would be more sensible to use the 2.x
series of Python.

 David, I think you are misunderstanding the purpose of HAVE_USABLE_WCHAR_T: 
 It does *not* specify whether wchar_t can be used. Instead, it specifies 
 whether wchar_t can be used as the datatype for Py_UNICODE. Calling it 
 HAVE_A_WCHAR_T_DEFINITION_THAT_IS_SUITABLE_FOR_USE_AS_BASE_TYPE_OF_PY_UNICODE 
 was just a little too tedious :-)

Haha :). Yes. My initial reading of the pyconfig.h was wrong. Got a
bit suspicious when my Linux box was not defining it. Then I saw them
memcpy and it made sense.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[Marc-Andre Lemburg]

Marc-Andre Lemburg m...@egenix.com added the comment:

David Coles wrote:
 
 I agree that it's not worthwhile trying to port Python to those Android 
 versions that have a single-byte wchar_t definition.
 
 Yup. Will be using Android 2.3+. If I'm forced to use an earlier
 version of Android I think it would be more sensible to use the 2.x
 series of Python.

Since you sound like you want to get Python running on Android,
are you aware of this project ?

  http://code.google.com/p/android-scripting/

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[David Coles]

David Coles coles.da...@gmail.com added the comment:

On Fri, May 6, 2011 at 3:17 PM, Marc-Andre Lemburg
rep...@bugs.python.org wrote:
 Since you sound like you want to get Python running on Android,
 are you aware of this project ?

  http://code.google.com/p/android-scripting/

Yes. It's excellent. I've actually been using it as a porting
reference for Python 3.

The problem is work decided to be very future proof with it's clients
and decided to use Python 3 as the embedded scripting language.
Because of differences in the C API, it looked like it might be easier
to do a port of Python 3 to Android (we already cross compile an ARM
version for another Linux platform) then to have the Android client
with 2.6 version.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[David Coles]

New submission from David Coles coles.da...@gmail.com:

On Android platforms bionic defines wchar_t as char. This causes compiling of 
unicodeobject.c to fail with #error unsupported wchar_t and PyUNICODE sizes, 
see issue #8670.

The unusual thing is that the configure script does detect if wchar_t is usable 
(HAVE_USABLE_WCHAR_T) but the wide code support block in unicodeobject.c does 
not check this (only an #ifdef HAVE_WCHAR_T).

Possibly the quick solution is to change this #ifdef to '#if 
defined(HAVE_WCHAR_T)  defined(HAVE_USABLE_WCHAR_T)'. The header 
unicodeobject.h does check for HAVE_USABLE_WCHAR_T but will only define 
HAVE_WCHAR_T if it is not already defined.

--
components: Build
messages: 135242
nosy: dcoles
priority: normal
severity: normal
status: open
title: Compile fails when sizeof(wchar_t) == 1
type: compile error

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Do you want to propose a patch? I'm not sure any of us can test on an Android 
setup.

--
nosy: +pitrou
stage:  - needs patch
versions: +Python 2.7, Python 3.1, Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[STINNER Victor]

Changes by STINNER Victor victor.stin...@haypocalc.com:


--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12010] Compile fails when sizeof(wchar_t) == 1

[David Coles]

David Coles coles.da...@gmail.com added the comment:

Sure. There are a few other build issues (mainly missing types/structs in 
Bionic) that are preventing a complete build. I'll put together a patch once I 
have a working build and can verify that HAVE_USABLE_WCHAR_T is set correctly 
set on a normal Linux build.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12010
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com