[issue16565] Increase Py_AddPendingCall array size

[Gregory P. Smith]

Changes by Gregory P. Smith :


--
versions: +Python 3.7 -Python 2.7, Python 3.3, Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16565] Increase Py_AddPendingCall array size

[Felipe Cruz]

Felipe Cruz added the comment:

Running test_aio_read
[New Thread 0x77ff7700 (LWP 20681)]
[New Thread 0x761ff700 (LWP 20682)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x761ff700 (LWP 20682)]
sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
34  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S: No such file or 
directory.
(gdb) backtrace
#0  sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
#1  0x0051b053 in PyThread_release_lock (lock=0x0) at 
Python/thread_pthread.h:346
#2  0x004c868f in PyEval_ReleaseLock () at Python/ceval.c:265
#3  0x00501974 in PyThreadState_DeleteCurrent () at Python/pystate.c:321
#4  0x00502116 in PyGILState_Release (oldstate=PyGILState_UNLOCKED) at 
Python/pystate.c:652
#5  0x7660ed44 in aio_completion_handler (sigval=...) at pyaio/core.c:26
#6  0x76409cdc in notify_func_wrapper (arg=0x78c0) at 
../sysdeps/pthread/aio_notify.c:45
#7  0x77bc4e9a in start_thread (arg=0x761ff700) at 
pthread_create.c:308
#8  0x771f14bd in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x in ?? ()

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16565
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16565] Increase Py_AddPendingCall array size

[Antoine Pitrou]

Antoine Pitrou added the comment:

Is it with a debug Python build? `PyThread_release_lock (lock=0x0)` is a bit 
worrying.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16565
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16565] Increase Py_AddPendingCall array size

[Felipe Cruz]

Felipe Cruz added the comment:

Yes! 2.7.3 build with pydebug.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16565
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16565] Increase Py_AddPendingCall array size

[Felipe Cruz]

Felipe Cruz added the comment:

Without debug backtrace

#0  sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
#1  0x004d456e in PyGILState_Release ()
#2  0x759b9cdc in notify_func_wrapper (arg=0x78c0) at 
../sysdeps/pthread/aio_notify.c:45
#3  0x77bc4e9a in start_thread (arg=0x759b5700) at 
pthread_create.c:308
#4  0x769b64bd in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x in ?? ()

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16565
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16565] Increase Py_AddPendingCall array size

[Jesús Cea Avión]

Changes by Jesús Cea Avión j...@jcea.es:


--
nosy: +jcea

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16565
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16565] Increase Py_AddPendingCall array size

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc added the comment:

This array would still have a fixed size.
Could you arrange your code so that you enqueue pending calls elsewhere, and 
call Py_AddPendingCall only once until the pending items have been processed?

--
nosy: +amaury.forgeotdarc

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16565
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16565] Increase Py_AddPendingCall array size

[Antoine Pitrou]

Antoine Pitrou added the comment:

I just took a look at 
https://github.com/felipecruz/pyaio/commit/127372ba0a6dbca4045256dcd653789ee35f6a78
 and it looks a bit silly to me: if the completion handler is called in a 
separate thread, then the completion handler can just take the GIL and put the 
results in a regular Python queue.

In short, using signals for any kind of high-speed I/O sounds completely 
archaic to me.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16565
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16565] Increase Py_AddPendingCall array size

[Felipe Cruz]

Felipe Cruz added the comment:

Just confirmed that signals is not a viable option. Is too slow, as Antonie 
already pointed. It's almost 5 times slower than with SIGEV_THREAD. 

The problem now is:

If I use Py_AddPendingCall, the tests can hang sometimes. I can still follow 
Amaury suggestion tough.

If I try to acquire the GIL PyGILState_Ensure() call PyObject_CallObject and 
release the GIL in the aio_completion_handler function a SEGFAULT occurs in 
2.7.3 but not in 3.3 and newer..

This branch works only on 3.3 and newer: 
https://github.com/felipecruz/pyaio/tree/feature/check_no_pending_call

This other branch will segfault 2.7.3 with just acquire and release GIL on the 
completion handler:
https://github.com/felipecruz/pyaio/tree/feature/py27_gil_error

Since this looks very strange, can someone confirm this behavior?

Tested on a Ubuntu12 64.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16565
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16565] Increase Py_AddPendingCall array size

[Antoine Pitrou]

Antoine Pitrou added the comment:

 If I try to acquire the GIL PyGILState_Ensure() call
 PyObject_CallObject and release the GIL in the aio_completion_handler
 function a SEGFAULT occurs in 2.7.3 but not in 3.3 and newer..

Well, can you post a traceback?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16565
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16565] Increase Py_AddPendingCall array size

[Felipe Cruz]

New submission from Felipe Cruz:

Current pending calls limit is too small and it can be easily reached in very 
intensive async file io applications.

There is a little hack in pyaio[1] which sleeps if Py_AddPendingCall returns  
0 but It's not totally clear to me why the size of pendind calls array is only 
32.


[1] https://github.com/felipecruz/pyaio

--
components: IO
messages: 176491
nosy: felipecruz
priority: normal
severity: normal
status: open
title: Increase Py_AddPendingCall array size
versions: Python 2.7, Python 3.3, Python 3.4, Python 3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16565
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com