[issue12010] Compile fails when sizeof(wchar_t) == 1
New submission from David Coles : On Android platforms bionic defines wchar_t as char. This causes compiling of unicodeobject.c to fail with "#error "unsupported wchar_t and PyUNICODE sizes, see issue #8670". The unusual thing is that the configure script does detect if wchar_t is usable (HAVE_USABLE_WCHAR_T) but the wide code support block in unicodeobject.c does not check this (only an #ifdef HAVE_WCHAR_T). Possibly the quick solution is to change this #ifdef to '#if defined(HAVE_WCHAR_T) && defined(HAVE_USABLE_WCHAR_T)'. The header unicodeobject.h does check for HAVE_USABLE_WCHAR_T but will only define HAVE_WCHAR_T if it is not already defined. -- components: Build messages: 135242 nosy: dcoles priority: normal severity: normal status: open title: Compile fails when sizeof(wchar_t) == 1 type: compile error ___ Python tracker <http://bugs.python.org/issue12010> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12010] Compile fails when sizeof(wchar_t) == 1
David Coles added the comment: Sure. There are a few other build issues (mainly missing types/structs in Bionic) that are preventing a complete build. I'll put together a patch once I have a working build and can verify that HAVE_USABLE_WCHAR_T is set correctly set on a normal Linux build. -- ___ Python tracker <http://bugs.python.org/issue12010> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12010] Compile fails when sizeof(wchar_t) == 1
David Coles added the comment: After doing some more investigation it appears that Android's wchar_t support before android-9 is totally broken (see http://android.git.kernel.org/?p=platform/ndk.git;a=blob_plain;f=docs/STANDALONE-TOOLCHAIN.html;hb=HEAD). With android-9 you get 4 byte wchar_t and working wide character functions. Possibly of more interest for Python is that it's no longer buildable without wchar_t support. While unicodeobject is pretty good at checking HAVE_WCHAR_H, a number of modules and even pythonrun.c directly use wchar_t or functions like PyUnicode_FromWideChar without providing a fallback. Does Python 3 now require wchar_t or are these bugs? (either option seems sensible). A few other notes: HAVE_USABLE_WCHAR_T looks like it was a check for unsigned/>16 bits wchar_t that would allow them to be directly memcpy'd. The code in unicodeobject.c seems not to really use this anymore except (it's happy with signed or unsigned) and it looks like the check is only used for Windows now. To properly support wchar_t of size 1 you would basically implement multibyte character storage either with UTF-8 or just packing two wchar_t's with UTF-16. At least in Android the distinction doesn't seem to matter as Android's internationalziation/localization policy seems to be "use Java". -- ___ Python tracker <http://bugs.python.org/issue12010> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12010] Compile fails when sizeof(wchar_t) == 1
David Coles added the comment: On Fri, May 6, 2011 at 1:31 PM, Marc-Andre Lemburg wrote: > wchar_t should be fairly portable these days. I think the main > problem is that we never assumed sizeof(wchar_t) == 1 to be a > possibility. On Windows, wchar_t was 16 bit and the glibc started > out with 32 bits. Well a 1 byte wchar_t is a bit "ass backwards". I think it's very much an edge case. :) > Note that HAVE_USABLE_WCHAR_T is only used to check whether > Python can use wchar_t as alias for Py_UNICODE. Python's Unicode > implementation needs Py_UNICODE to be an unsigned type with > either 2 bytes or 4 bytes. If wchar_t does not provide these > sizes or is a signed type, Python cannot use it for Py_UNICODE > and must instead use "unsigned short". Right. That makes sense. In that case it's probably sensible to keep around. > If the configure script does not detect this case, then a patch > would be helpful. Yup. I'll put something together that causes configure to bail out if you're either missing HAVE_WCHAR_H or if SIZEOF_WCHAR_T is less than 16 bits. > Python should not use wchar_t for Py_UNICODE on such platforms > and instead go with "unsigned short". > > I would assume that the wchar_t C lib routines work based on UTF-8 > with sizeof(wchar_t) == 1, so the PyUnicode_*WideChar*() APIs would > need to be adjusted to work more or less like the UTF-8 codecs. Yes. Using UTF-8 would be the sensible solution. Sadly it looks like all the wide character functions <2.3 are undefined, so in this case Android saying it has wchar_t support is worse than useless. On Fri, May 6, 2011 at 1:37 PM, Marc-Andre Lemburg wrote: > With none of the wide-char functions working in Android <2.3, I don't > think you have a good chance of getting Python 3.x working, unless > you remove all their uses in the code and replace them with standard > char* functions. I agree. In my case I should be able to bump the required version number without too much fuss. It seems a bit silly to write in support for a platform that no longer supports said feature. > The last paragraph doesn't sound very promising either. I wonder > what they mean with "better representation". The C standard doesn't > have any better representation for Unicode at the moment. In C I guess the only sensible alternative would be UTF-8 char strings (or maybe using uint32_t), but in Python's case it really depends on how the underlying OS represents internationalized characters. Perhaps in other projects you would use an external library like ICU, but that's out the scope of my experience. :) -- ___ Python tracker <http://bugs.python.org/issue12010> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12010] Compile fails when sizeof(wchar_t) == 1
David Coles added the comment: On Fri, May 6, 2011 at 2:24 PM, Martin v. Löwis wrote: > > Martin v. Löwis added the comment: > > I think what they mean is a better representation from an Android API, such > as UChar32 from utils/AndroidUnicode.h. Ah. Sadly I don't think that's exposed in the NDK yet. > I agree that it's not worthwhile trying to port Python to those Android > versions that have a single-byte wchar_t definition. Yup. Will be using Android 2.3+. If I'm forced to use an earlier version of Android I think it would be more sensible to use the 2.x series of Python. > David, I think you are misunderstanding the purpose of HAVE_USABLE_WCHAR_T: > It does *not* specify whether wchar_t can be used. Instead, it specifies > whether wchar_t can be used as the datatype for Py_UNICODE. Calling it > HAVE_A_WCHAR_T_DEFINITION_THAT_IS_SUITABLE_FOR_USE_AS_BASE_TYPE_OF_PY_UNICODE > was just a little too tedious :-) Haha :). Yes. My initial reading of the pyconfig.h was wrong. Got a bit suspicious when my Linux box was not defining it. Then I saw them memcpy and it made sense. -- ___ Python tracker <http://bugs.python.org/issue12010> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12010] Compile fails when sizeof(wchar_t) == 1
David Coles added the comment: On Fri, May 6, 2011 at 3:17 PM, Marc-Andre Lemburg wrote: > Since you sound like you want to get Python running on Android, > are you aware of this project ? > > http://code.google.com/p/android-scripting/ Yes. It's excellent. I've actually been using it as a porting reference for Python 3. The problem is work decided to be very future proof with it's clients and decided to use Python 3 as the embedded scripting language. Because of differences in the C API, it looked like it might be easier to do a port of Python 3 to Android (we already cross compile an ARM version for another Linux platform) then to have the Android client with 2.6 version. -- ___ Python tracker <http://bugs.python.org/issue12010> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12010] Compile fails when sizeof(wchar_t) == 1
David Coles added the comment: Attached is a patch that updates configure.in to make sure that wchar.h is present and that wchar_t is at least 16 bits wide. On android-8 this patch causes the configure step to fail since SIZEOF_WCHAR_T == 1. On android-9 and my Linux desktop the build continues as normal, passing the build tests. If wchar.h is removed from the system then the build also fails with an error as expected. The patch does not contain the configure diff since it also contained some other changes and I wasn't sure of the correct autoreconf process for Python. -- keywords: +patch Added file: http://bugs.python.org/file22012/wchar_check.patch ___ Python tracker <http://bugs.python.org/issue12010> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38897] Example in socket documentation uses deprecated array.fromstring
New submission from David Coles : See the `recv_fds` example for `socket.recvmsg`. This code produces a `DeprecationWarning` on current versions of Python. https://docs.python.org/3.9/library/socket.html#socket.socket.recvmsg -- assignee: docs@python components: Distutils, Documentation messages: 357314 nosy: corona10, dcoles, docs@python, dstufft, eric.araujo priority: normal pull_requests: 16840 severity: normal status: open title: Example in socket documentation uses deprecated array.fromstring versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue38897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19308] Tools/gdb/libpython.py does not support GDB linked against Python 3
New submission from David Coles: Tools/gdb/libpython.py is currently Python 3 incompatible. Unfortunately recent versions of gdb (such as the one provided in Ubuntu 13.10) may be linked against Python 3 rather than Python 2, breaking debugging support. Most of the issues appear to be trivial issues such as the print statement syntax and removal of functions like xrange, unichr and long. -- components: Demos and Tools messages: 200554 nosy: dcoles priority: normal severity: normal status: open title: Tools/gdb/libpython.py does not support GDB linked against Python 3 versions: Python 3.3, Python 3.4 ___ Python tracker <http://bugs.python.org/issue19308> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19308] Tools/gdb/libpython.py does not support GDB linked against Python 3
David Coles added the comment: Sure thing. I've got a patch attached to https://bugs.launchpad.net/ubuntu/+source/gdb/+bug/1241668 but want to make sure that it doesn't break things on the py2-linked version. -- ___ Python tracker <http://bugs.python.org/issue19308> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19308] Tools/gdb/libpython.py does not support GDB linked against Python 3
David Coles added the comment: I ended up with a very similar looking patch - and so merged my changes into Pitrou's patch. I also had a go at fixing up the `test_gdb` unit test, which revealed a few more string/unicode issues. Finally tracked them down to the `write_unicode` function. Sadly the `write` function in GDB's Python API only accepts unicode strings, hence we must pre-backslash escape for C-string encoding (either Python's default encoding or utf-8 in Python 3.1+) then turn it back into a unicode string again to avoid a TypeError. Yuck. The patch almost applies on Python 2.7, but needs a little bit of massaging due to slight differences in Tools/gdb/libpython.py. Need to test the new unicode changes haven't broken anything and then I'll upload a patch for Python 2.7. -- Added file: http://bugs.python.org/file32271/gdb_py3.patch ___ Python tracker <http://bugs.python.org/issue19308> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17772] test_gdb doesn't detect a gdb built with python3.3 (or higher)
David Coles added the comment: Attached is a patch that enables the test for gdb linked against py3k. All test failures should be fixed by the patch on issue19308. -- keywords: +patch nosy: +dcoles Added file: http://bugs.python.org/file32272/test_gdb-py3k-compat.patch ___ Python tracker <http://bugs.python.org/issue17772> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17772] test_gdb doesn't detect a gdb built with python3.3 (or higher)
David Coles added the comment: Should probably also be applied to Python 2.7 branch since it's possible be debugging Python 2.7 with a version of py3k-linked gdb. -- versions: +Python 2.7 ___ Python tracker <http://bugs.python.org/issue17772> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19308] Tools/gdb/libpython.py does not support GDB linked against Python 3
David Coles added the comment: And here's the patch for Python 2.7. The result of testing is as follows: - python (default) against py2-linked gdb: All tests pass - python (2.7) against py2-linked gdb: `test_long` fails. - python (default) against py3-linked gdb: All tests pass - python (2.7) against py3-linked gdb: `test_long`, `test_strings` and `test_unicode` fail. `test_long` has been removed from the default branch, in addition gdb doesn't seem to distinguish printing 0 from 0L. `test_strings` likely fails since Python 3 will print printable unicode characters (like "\xff") while Python 2 will not. `test_unicode` likely fails due to the 'u' prefix on unicode strings in Python 2. This kind of behavior is pretty much expected when cross-testing gdb. -- Added file: http://bugs.python.org/file32273/python27-gdb_py3.patch ___ Python tracker <http://bugs.python.org/issue19308> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22970] Cancelling wait() after notification leaves Condition in an inconsistent state
New submission from David Coles:
If a task that is waiting on an asyncio.Condition is cancelled after
notification but before having successfully reacquired the associated lock, the
acquire() will be cancelled causing wait() to return without the lock held
(violating wait()'s contract and leaving the program in an inconsistent state).
This can be reproduced in cases where there's some contention on the
Condition's lock. For example:
import asyncio
loop = asyncio.get_event_loop()
cond = asyncio.Condition()
@asyncio.coroutine
def cond_wait_timeout(condition, timeout):
wait_task = asyncio.async(condition.wait())
loop.call_later(timeout, wait_task.cancel)
try:
yield from wait_task
return True
except asyncio.CancelledError:
print("Timeout (locked: {0})".format(condition.locked()))
return False
@asyncio.coroutine
def waiter():
yield from cond.acquire()
try:
print("Wait")
if (yield from cond_wait_timeout(cond, 1)):
# Cause some lock contention
print("Do work")
yield from asyncio.sleep(1)
finally:
cond.release()
@asyncio.coroutine
def notifier():
# Yield to the waiters
yield from asyncio.sleep(0.1)
yield from cond.acquire()
try:
print("Notify")
cond.notify_all()
finally:
cond.release()
loop.run_until_complete(asyncio.wait([waiter(), waiter(), notifier()]))
The most straightforward fix appears to be just to have wait() retry to acquire
the lock, effectively ignoring cancellation at this point (since the condition
has already finished waiting and just trying to reacquire the lock before
returning).
--
components: asyncio
files: asyncio-fix-wait-cancellation-race.patch
keywords: patch
messages: 231912
nosy: dcoles, gvanrossum, haypo, yselivanov
priority: normal
severity: normal
status: open
title: Cancelling wait() after notification leaves Condition in an inconsistent
state
type: behavior
versions: Python 3.4
Added file:
http://bugs.python.org/file37330/asyncio-fix-wait-cancellation-race.patch
___
Python tracker
<http://bugs.python.org/issue22970>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22970] Cancelling wait() after notification leaves Condition in an inconsistent state
David Coles added the comment: Hi Victor, (Sorry for the delay, I think GMail ate the notification) The main issue is that Condition.wait MUST reacquire the associated lock released on entry to wait() before returning or else the caller's assumption that it holds a lock (such as `with lock:`) will be incorrect. With the threading module, it's typically not possible to interrupt or cancel lock acquisition (no Thread.interrupt for example). Even on Python 3.2 where lock acquisition can be interrupted by POSIX signals, Condition.wait's implementation uses the non-interruptable lock._acquire_restore (calls rlock_acquire_restore → PyThread_acquire_lock → PyThread_acquire_lock → PyThread_acquire_lock_timed with intr=0 [Python/thread_pthread.h] which does the uninterruptable retry loop). So, not a problem for threading.Condition.wait(). As for re-raising the CancelledError after the lock is reacquired, I'm not sure it serves any useful purpose to the caller since by this point we're either in the process of waking up after a notify() or already raising an exception (such as a CancelledError if someone called task.cancel() _before_ the condition was notified). Making the caller also handle re-acquisition failure would be quite messy. For example, one idea would be to have the caller check cond.locked() after a CancelledError and have the caller reacquire, but it turns in asyncio you can't tell whom owns the lock (could have been grabbed by another task). -- ___ Python tracker <http://bugs.python.org/issue22970> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22970] Cancelling wait() after notification leaves Condition in an inconsistent state
David Coles added the comment: Just for context, the reason for using a custom wait function (cond_wait_timeout) rather than the more idiomatic asyncio.wait_for(cond.wait(), timeout) is that the combination introduces another subtle, but nasty locking inconsistency (see https://groups.google.com/forum/#!topic/python-tulip/eSm7rZAe9LM ). Should probably open a ticket for that issue too. -- ___ Python tracker <http://bugs.python.org/issue22970> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22970] asyncio: Cancelling wait() after notification leaves Condition in an inconsistent state
David Coles added the comment: Since I reported this, I haven't seen any proposed solutions other other than a retry loop to ensure that the lock is guaranteed to be reacquired when the sleeping coroutine is woken. The four possibilities of cancelling the waiting coroutine are: 1. Before wait is scheduled (Because the coroutine is never run, the lock is never released and a CancelledError is thrown by the runtime) 2. While waiting to be notified (CancelledError is thrown, but the 'finally' clause reacquires lock) 3. After wait has returned (This is a no-op since the wait coroutine has already completed - no CancelledError is thrown) 4. After notification, but while waiting on lock reaquisition (The problem discussed here) Cancellation is quite analogous to EINTER of system calls interrupted by signals - you'll notice that the CPython runtime will retry acquiring the lock forever https://hg.python.org/cpython/file/default/Python/thread_pthread.h#l355 . This is acceptable behaviour acording the the POSIX spec http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_cond_timedwait.html , in particular: "If a signal is delivered to a thread waiting for a condition variable, upon return from the signal handler the thread resumes waiting for the condition variable as if it was not interrupted, or it shall return zero due to spurious wakeup." "Upon successful return, the mutex shall have been locked and shall be owned by the calling thread." "These functions shall not return an error code of [EINTR]." Thus, other than throwing something like a RuntimeError, there's no way to legally return from a Condition.wait without having that lock reacquired. Thus the only option is to retry if the lock reacquire is interrupted. -- ___ Python tracker <http://bugs.python.org/issue22970> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22970] asyncio: Cancelling wait() after notification leaves Condition in an inconsistent state
David Coles added the comment: Hi Yury, Sure - I'll create a PR along with a test that reproduces the issue. Should be able to get that done this weekend. -- ___ Python tracker <http://bugs.python.org/issue22970> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22970] asyncio: Cancelling wait() after notification leaves Condition in an inconsistent state
David Coles added the comment: Please find the PR including a test to reproduce the issue here: https://github.com/python/asyncio/pull/346 -- ___ Python tracker <http://bugs.python.org/issue22970> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22970] Cancelling wait() after notification leaves Condition in an inconsistent state
David Coles added the comment: This issue can still be reproduced on Python 3.5.0a1. (Triggers a "RuntimeError: Lock is not acquired" on cond.release()) Please let me know if there's any further steps you'd like me to take here. -- versions: +Python 3.5 ___ Python tracker <http://bugs.python.org/issue22970> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
