[issue4028] Problem compiling the multiprocessing module on sunos5
Charles-François Natali neolo...@free.fr added the comment:
Hello,
there's some issues compiling the multiprocessing module on the SunOS
I have here, where CMSG_LEN, CMSG_ALIGN, CMSG_SPACE and sem_timedwait
are absent.
CMSG_LEN and friends should be defined by sys/socket.h (as required by
POSIX). SunOS 5.10 man page lists them:
http://download.oracle.com/docs/cd/E19253-01/816-5173/socket.h-3head/index.html
But not the SunOS 5.9 version:
http://ewp.rpi.edu/hartford/webgen/sysdocs/C/solaris_9/SUNWaman/hman3head/socket.3head.html
it looks like simply defining the first three macros like this works
It works, but it's probably not a good idea: if the headers don't define
CMSG_LEN and friends, then FD passing will probably not work.
It'd be better to not compile multiprocessing_(sendfd|recvfd) if CMSG_LEN is
not defined (patch attached).
sem_timedwait are absent.
Hmmm.
Do you have the compilation's log?
Normally, if sem_timedwait isn't available, HAVE_SEM_TIMEDWAIT shouldn't be
defined, and we should fallback to sem_trywait (by the way, calling sem_trywait
multiple times until the timeout expires is not the same has calling
sem_timedwait: this will fail in case of heavy contention).
So this should build correctly.
And this seems even stranger when I read Sebastian's message:
so I had to commented out HAVE_SEM_TIMEDWAIT from setup.py, see:
elif platform.startswith('sunos5'):
macros = dict(
HAVE_SEM_OPEN=1,
HAVE_FD_TRANSFER=1
)
#HAVE_SEM_TIMEDWAIT=0,
libraries = ['rt']
Makes sense.
If we define HAVE_SEMTIMEDWAIT=0, then code guarded by
#ifdef HAVE_SEMTIMEDWAIT
will be compiled, and the linker won't be able to resolve sem_timedwait.
The preprocessor just checks that the symbol is defined, not that it has a
non-zero value.
To sum up: could someone with a SunOS box test the attached patch, and post the
compilation logs if it still fails?
--
keywords: +patch
nosy: +neologix
Added file: http://bugs.python.org/file23058/multiprocessing_sendfd.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4028
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12287] ossaudiodev: stack corruption with FD = FD_SETSIZE
Charles-François Natali neolo...@free.fr added the comment: Alright, committed to 2.7, 3.2 an default. Seems to work on all the buildbots, closing. -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12287 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12801] C realpath not used by os.path.realpath
Charles-François Natali neolo...@free.fr added the comment: Well, if we use two different paths based on the libc version, it might not be a good idea, since behaviour can be different in some cases. Indeed. It would be nice to know if some modern platforms have a non-compliant realpath(). Alas, it doesn't seem to hold for OpenBSD: http://old.nabble.com/Make-realpath(3)-conform-to-SUSv4-td32031895.html A patch supporting NULL was committed two months ago, which means we probably can't push this forward. I've been quite disappointed by POSIX lately... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12801 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12801] C realpath not used by os.path.realpath
Charles-François Natali neolo...@free.fr added the comment: POSIX the standard, or the implementers?? Both :-) For those wondering why we can't use PATH_MAX (ignoring the buffer overallocation), here's why: https://www.securecoding.cert.org/confluence/display/cplusplus/FIO02-CPP.+Canonicalize+path+names+originating+from+untrusted+sources Avoid using this function. It is broken by design since (unless using the non-standard resolved_path == NULL feature) it is impossible to determine a suitable size for the output buffer, resolved_path. According to POSIX a buffer of size PATH_MAX suffices, but PATH_MAX need not be a defined constant, and may have to be obtained using pathconf(3). And asking pathconf(3) does not really help, since on the one hand POSIX warns that the result of pathconf(3) may be huge and unsuitable for mallocing memory. And on the other hand pathconf(3) may return -1 to signify that PATH_MAX is not bounded. The libc4 and libc5 implementation contains a buffer overflow (fixed in libc-5.4.13). As a result, set-user-ID programs like mount(8) need a private version. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12801 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12852] test_posix.test_fdlistdir() segfault on OpenBSD
Charles-François Natali neolo...@free.fr added the comment: [Switching to process 21658, thread 0x20a519000] _readdir_unlocked (dirp=0xafb0e80, result=0x7f7d7ac0, skipdeleted=1) at /usr/src/lib/libc/gen/readdir.c:44 44 if (dirp-dd_loc = dirp-dd_size) Looks like dirp points to an invali location in memory. Could you try display it (p *dirp)? But this definitely looks like a kernel/libc bug... -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12852 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12852] test_posix.test_fdlistdir() segfault on OpenBSD
Charles-François Natali neolo...@free.fr added the comment: I think that the problem is that fdopendir() is not defined. If a function is not defined, C uses int as the result type. An int is not enough to store a 64-bit pointer. See in gdb output: dirp is 0x0afb0e80 whereas other pointers look like 0x20973fc30. You missed the highest hexa digit (0x2). Yeah, I noticed that. I didn't make the connection with the possibility of missing prototype though. Nice catch. I tried AC_DEFINE(_POSIX_C_SOURCE, 200809L, Define to activate features from IEEE Stds 1003.1-2008) but it doesn't work. You mean that the patch you attached doesn't work, correct? I know it's a stupid question, but you're sure you didn't forget to run autoconf/autoheader? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12852 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1462440] socket and threading: udp multicast setsockopt fails
Changes by Charles-François Natali neolo...@free.fr: -- resolution: - invalid stage: needs patch - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1462440 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12868] test_faulthandler.test_stack_overflow() failed on OpenBSD
Charles-François Natali neolo...@free.fr added the comment: OpenBSD's threads are userland threads, and sigaltstack() doesn't work when the program is built with -pthread: http://marc.info/?l=openbsd-bugsm=114323355014696w=2 Note that POSIX warns about this: http://www.opengroup.org/onlinepubs/95399/functions/sigaltstack.html Use of this function by library threads that are not bound to kernel-scheduled entities results in undefined behavior. I think we should skip this test on OpenBSD when Python is compiled with threads support. Out of curiosity, could you try this: $ ./python -c import faulthandler; faulthandler.enable(); faulthandler._stack_overflow(); echo $? And if you're motivated, you could try it again after having built python with './configure --without-threads'. -- components: +Tests nosy: +haypo, neologix stage: - needs patch type: - behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12868 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12868] test_faulthandler.test_stack_overflow() failed on OpenBSD
Changes by Charles-François Natali neolo...@free.fr: -- keywords: +patch Added file: http://bugs.python.org/file23078/openbsd_sigaltstack.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12868 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12871] Disable sched_get_priority_min/max if Python is compiled without threads
Charles-François Natali neolo...@free.fr added the comment:
There's no reason to disable sched_get_priority_(min|max) when Python is built
without threads: those libraries control the scheduling policy, and should be
available even without pthread.
However, it's really likely that pthread has its own implementation (especially
since OpenBSD's threads are implemented in user-space), and it seems that
OpenBSD sched_get_priority() is only defined for threads:
http://www.openbsd.org/cgi-bin/man.cgi?query=sched_get_priority_maxapropos=0sektion=0manpath=OpenBSD+Currentarch=i386format=html
This implementation does not support process scheduling.
For example, the following snippet builds correctly on Linux:
#include sched.h
int main(int argc, char *argv[])
{
int (*fp)(int) = sched_get_priority_max;
return 0;
}
So this should be skipped only on OpenBSD, or we should add some checks to the
configure script.
--
nosy: +neologix
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12871
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12868] test_faulthandler.test_stack_overflow() failed on OpenBSD
Charles-François Natali neolo...@free.fr added the comment: It does not build completely, I have a problem if I add --without-threads: Until this gets fixed, if you want to do a quick test, you could just remove the calls to sched_get_priority_(min|max): diff -r 0968acf0e6db Modules/posixmodule.c --- a/Modules/posixmodule.c Wed Aug 31 16:52:12 2011 +0200 +++ b/Modules/posixmodule.c Wed Aug 31 22:51:13 2011 +0200 @@ -4562,7 +4562,6 @@ if (!PyArg_ParseTuple(args, i:sched_get_priority_max, policy)) return NULL; -max = sched_get_priority_max(policy); if (max 0) return posix_error(); return PyLong_FromLong(max); @@ -4579,7 +4578,6 @@ if (!PyArg_ParseTuple(args, i:sched_get_priority_min, policy)) return NULL; -min = sched_get_priority_min(policy); if (min 0) return posix_error(); return PyLong_FromLong(min); That should allow you to rebuild with 'without-threads'. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12868 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Charles-François Natali neolo...@free.fr added the comment: Anyway, since my view does not seem to resonate with core developers I I'll give it a rest for now. Well, the problem is that many views have been expressed in this thread, which doesn't help getting a clear picture of what's needed to make progress on this issue. AFAIC, I think the following seems reasonable: 1) add an atfork module which provides a generic and pthread_atfork-like mechanism to setup handlers that must be called after fork (right now several modules use their own ad-hoc mechanism) 2) for multiprocessing, call exec() after fork() (issue #8713) 3) for buffered file objects locks, use the approach similar to the patch I posted (reinit locks in the child process right after fork()) Does that sound reasonable? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12472] Build failure on IRIX
Charles-François Natali neolo...@free.fr added the comment: I'm closing, since IRIX header files seem terminally broken, and we can't do much about it. Furthermore, I'm 99% sure IRIX isn't officially supported anymore. -- resolution: - wont fix stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12472 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12837] Patch for issue #12810 removed a valid check on socket ancillary data
Charles-François Natali neolo...@free.fr added the comment: `long long` is not ANSI, but C99. Anyhow, I'm still not sure this check is necessary, because: 1) I really doubt any modern OS uses a signed socklen_t 2) even if it's the case, I don't see how we could possibly end up with a negative msg_controllen I'm adding Nick and Antoine to the noisy list to know what they think about this... -- nosy: +ncoghlan, pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12837 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12868] test_faulthandler.test_stack_overflow() failed on OpenBSD
Charles-François Natali neolo...@free.fr added the comment: without-threads, it segfault: It's normal :-) _stack_overflow triggers - as it names implies - a stack overflow. However, as you can see in the output, faulthandler is now able to catch the SIGSEGV and display the backtrace (because it set up an alternate stack for the signal handler with sigaltstack). However, if I run test_faulthandler.py, it seems to be ok: Yes: the test checks that the stack overflow was correctly caught by faulthandler. @Victor: can I commit the patch? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12868 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12868] test_faulthandler.test_stack_overflow() failed on OpenBSD
Charles-François Natali neolo...@free.fr added the comment: Here's a patch with an updated skip message. As for rthreads support, a quick search seems to indicate that its API is exactly the same as pthreads, and it's even binary compatible. Python will automatically use it when run on a OpenBSD system with rthreads enabled. -- Added file: http://bugs.python.org/file23082/openbsd_sigaltstack-1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12868 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12868] test_faulthandler.test_stack_overflow() failed on OpenBSD
Changes by Charles-François Natali neolo...@free.fr: Removed file: http://bugs.python.org/file23078/openbsd_sigaltstack.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12868 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12868] test_faulthandler.test_stack_overflow() failed on OpenBSD
Charles-François Natali neolo...@free.fr added the comment: Committed. Rémi, thanks once again for this report! -- resolution: - fixed stage: needs patch - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12868 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12871] Disable sched_get_priority_min/max if Python is compiled without threads
Charles-François Natali neolo...@free.fr added the comment: It builds correctly with -pthread or -lpthread, but it fails to build without these options. Not on Linux: this is specific to OpenBSD. sched_get_priority_max() and sched_get_priority_min() come from libpthread on OpenBSD, whereas Python only checks for #ifdef HAVE_SCHED_H. That's exactly what I said in my previous message ;-) I'll post a patch later. -- title: Disable sched_get_priority_min/max on OpenBSD if Python is compiled without threads - Disable sched_get_priority_min/max if Python is compiled without threads ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12871 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12882] mmap crash on Windows
Changes by Charles-François Natali neolo...@free.fr: -- Removed message: http://bugs.python.org/msg143397 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12882 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12871] Disable sched_get_priority_min/max if Python is compiled without threads
Charles-François Natali neolo...@free.fr added the comment:
Here's a patch adding a configure-time check. Since the functions are
checked without being linked explicitely with pthread, it should do
the trick (I couldn't test it on OpenBSD though).
I also added a skipTest to test_posix.test_sched_priority().
--
keywords: +patch
title: Disable sched_get_priority_min/max if Python is compiled without threads
- Disable sched_get_priority_min/max if Python is compiled without threads
Added file: http://bugs.python.org/file23097/sched_get_priority.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12871
___diff -r a29b72950795 Lib/test/test_posix.py
--- a/Lib/test/test_posix.pyThu Sep 01 23:08:21 2011 +0200
+++ b/Lib/test/test_posix.pySat Sep 03 18:08:01 2011 +0200
@@ -840,6 +840,8 @@
posix.sched_yield()
@requires_sched_h
+@unittest.skipUnless(hasattr(posix, 'sched_get_priority_max'),
+ requires sched_get_priority_max())
def test_sched_priority(self):
# Round-robin usually has interesting priorities.
pol = posix.SCHED_RR
diff -r a29b72950795 Modules/posixmodule.c
--- a/Modules/posixmodule.c Thu Sep 01 23:08:21 2011 +0200
+++ b/Modules/posixmodule.c Sat Sep 03 18:08:01 2011 +0200
@@ -4555,6 +4555,8 @@
#ifdef HAVE_SCHED_H
+#ifdef HAVE_SCHED_GET_PRIORITY_MAX
+
PyDoc_STRVAR(posix_sched_get_priority_max__doc__,
sched_get_priority_max(policy)\n\n\
Get the maximum scheduling priority for *policy*.);
@@ -4589,6 +4591,8 @@
return PyLong_FromLong(min);
}
+#endif /* HAVE_SCHED_GET_PRIORITY_MAX */
+
#ifdef HAVE_SCHED_SETSCHEDULER
PyDoc_STRVAR(posix_sched_getscheduler__doc__,
@@ -10452,8 +10456,10 @@
{fork,posix_fork, METH_NOARGS, posix_fork__doc__},
#endif /* HAVE_FORK */
#ifdef HAVE_SCHED_H
+#ifdef HAVE_SCHED_GET_PRIORITY_MAX
{sched_get_priority_max, posix_sched_get_priority_max, METH_VARARGS,
posix_sched_get_priority_max__doc__},
{sched_get_priority_min, posix_sched_get_priority_min, METH_VARARGS,
posix_sched_get_priority_min__doc__},
+#endif
#ifdef HAVE_SCHED_SETPARAM
{sched_getparam, posix_sched_getparam, METH_VARARGS,
posix_sched_getparam__doc__},
#endif
@@ -10474,7 +10480,7 @@
{sched_setaffinity, posix_sched_setaffinity, METH_VARARGS,
posix_sched_setaffinity__doc__},
{sched_getaffinity, posix_sched_getaffinity, METH_VARARGS,
posix_sched_getaffinity__doc__},
#endif
-#endif
+#endif /* HAVE_SCHED_H */
#if defined(HAVE_OPENPTY) || defined(HAVE__GETPTY) || defined(HAVE_DEV_PTMX)
{openpty, posix_openpty, METH_NOARGS, posix_openpty__doc__},
#endif /* HAVE_OPENPTY || HAVE__GETPTY || HAVE_DEV_PTMX */
diff -r a29b72950795 configure.in
--- a/configure.in Thu Sep 01 23:08:21 2011 +0200
+++ b/configure.in Sat Sep 03 18:08:01 2011 +0200
@@ -2538,7 +2538,8 @@
select sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid
seteuid \
setgid sethostname \
setlocale setregid setreuid setresuid setresgid setsid setpgid setpgrp
setpriority setuid setvbuf \
- sched_setaffinity sched_setscheduler sched_setparam sched_rr_get_interval \
+ sched_get_priority_max sched_setaffinity sched_setscheduler sched_setparam \
+ sched_rr_get_interval \
sigaction sigaltstack siginterrupt sigpending sigrelse \
sigtimedwait sigwait sigwaitinfo snprintf strftime strlcpy symlinkat sync \
sysconf tcgetpgrp tcsetpgrp tempnam timegm times tmpfile tmpnam tmpnam_r \
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12905] multiple errors in test_socket on OpenBSD
Charles-François Natali neolo...@free.fr added the comment: I hope that this issue is not related to threads+signals. We got many threads+signals issues on FreeBSD 6. Yep. OpenBSD has a really specific pthread implementation (in user-space, using non-blocking I/O), so it might very well be yet another threads+signals occurrence. @Rémi What happens if you run a code equivalent to test_sendall_interrupted on its own? I mean, if you try something like this: import signal, socket c, s = socket.socketpair() signal.signal(signal.SIGALRM, lambda x,y: 0) signal.alarm(1) c.sendall(bx * (1024**2)) If it works, you could try creating a dummy thread before setting up the signal handler, something like: t = threading.Thread(target=lambda: 0) t.start() t.join() And retry. The problem with all those interruption tests (like issue #12903) is that they'll break on many *BSD if another thread is running. Although in this specific case, I suspect that there are no threads running, and we're really encountering a kernel/libc bug (Victor, remember the funny bug on FreeBSD where kill(getpid(), signal) didn't deliver the signal synchronously after the first thread had been created?). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12905 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12871] Disable sched_get_priority_min/max if Python is compiled without threads
Changes by Charles-François Natali neolo...@free.fr: -- resolution: - fixed stage: - committed/rejected status: open - closed type: - compile error ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12871 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12905] multiple errors in test_socket on OpenBSD
Charles-François Natali neolo...@free.fr added the comment: Hi, it blocks too: Oops, I just realized there was a typo in the sample test. The signal handler should be lambda x,y: 1/0 and not lambda x,y: 0 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12905 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12905] multiple errors in test_socket on OpenBSD
Charles-François Natali neolo...@free.fr added the comment: The C signal handler is called, but the system call (read in this case) is not interrupted. That's what I thought... Bad news: the script doesn't hang if Python is build without threads. Makes sense. When linked with pthread, all I/O syscalls are actually non-blocking. read() is interrupted after 1 second, it works. Hmmm... Does it still work if you don't a create thread beforehand? Also, one difference is that Python uses sigaction to setup the signal handler. There might be subtle semantics change/bugs between signal/sigaction. Oh, siginterrupt(SIGALRM, 0) doesn't work in a program linked to pthread. You could try with sigaction/SA_RESTART. But OpenBSD's pthread implementation has severe limitations/bugs. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12905 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12905] multiple errors in test_socket on OpenBSD
Charles-François Natali neolo...@free.fr added the comment: Using SA_RESTART, read() is not interrupted. But if the program is linked to pthread, read() is always interrupted: with sa_flags=0 or sa_flags=SA_RESTART. Ouch... But OpenBSD's pthread implementation has severe limitations/bugs. rthread doc contains: Future work: Quite simply, signal handling is one the most complicated aspects of threads to get right. (...) This paper dates back to 2005, I was hoping they would have solved this by now... As for the original problem, IIUC you don't reproduce it with your C test code... It might be due to a subtle difference in the way Python is built (like POSIX_SOURCE...), but it's hard to tell... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12905 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12852] POSIX level issues in posixmodule.c on OpenBSD 5.0
Charles-François Natali neolo...@free.fr added the comment: this is the result of gcc -E on Modules/posixmodule.o, asked by haypo. And this confirms that __POSIX_VISIBLE 200809 when dirent.h is included, hence the missing prototype. I suppose that there is a conflict between Python's _POSIX_C_SOURCE and other defines related to the POSIX level. Lookie here: http://www.openbsd.org/cgi-bin/cvsweb/src/sys/sys/cdefs.h?rev=1.31 #ifdef _XOPEN_SOURCE # if (_XOPEN_SOURCE - 0 = 700) # define __XPG_VISIBLE 700 # undef _POSIX_C_SOURCE # define _POSIX_C_SOURCE 200809L # elif (_XOPEN_SOURCE - 0 = 600) # define __XPG_VISIBLE 600 # undef _POSIX_C_SOURCE # define _POSIX_C_SOURCE 200112L configure.in defines _XOPEN_SOURCE to 600: if test $define_xopen_source = yes then AC_DEFINE(_XOPEN_SOURCE, 600, Define to the level of X/Open that your system supports) So, try with _POSIX_C_SOURCE set to 200809L and _XOPEN_SOURCE to 700 (see http://pubs.opengroup.org/onlinepubs/9699919799/), and it should work. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12852 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12852] POSIX level issues in posixmodule.c on OpenBSD 5.0
Charles-François Natali neolo...@free.fr added the comment: _POSIX_C_SOURCE value is set automatically depending on _XOPEN_SOURCE value. I know, but I think it's better to be consistent an also bump _POSIX_C_SOURCE to POSIX 2008, and follow POSIX's recommandation (http://pubs.opengroup.org/onlinepubs/9699919799/): A Strictly Conforming POSIX Application is an application that requires only the facilities described in POSIX.1-2008. Such an application: [...] For the C programming language, shall define _POSIX_C_SOURCE to be 200809L before any header is included -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12852 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12905] multiple errors in test_socket on OpenBSD
Charles-François Natali neolo...@free.fr added the comment: It looks like Python cannot do much to workaround OpenBSD issues. IMO the best fix is just to skip these tests on OpenBSD, until OpenBSD handles correctly signals in programs linked to pthread. The same fix can be used for #12903. Agreed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12905 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12936] armv5tejl: random segfaults in getaddrinfo()
Charles-François Natali neolo...@free.fr added the comment: You don't have a core dump, do you? -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12936 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12936] armv5tejl: random segfaults in getaddrinfo()
Charles-François Natali neolo...@free.fr added the comment: No such luck. Somehow gdb doesn't dump the core file: What do $ /sbin/sysctl -a | grep kernel.core And $ grep core /etc/security/limits.conf return? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12936 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12936] armv5tejl: random segfaults in getaddrinfo()
Charles-François Natali neolo...@free.fr added the comment: Traceback with faulthandler disabled: It crashes when trying to look up TLS (which explains why it doesn't crash when built ``without-threads`). Looks like a libc bug, but would it be possible to have a backtrace with Python built with `with-pydebug`? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12936 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12936] armv5tejl: random segfaults in getaddrinfo()
Charles-François Natali neolo...@free.fr added the comment: Could faulthandler cause problems like these: Well, that would explain why it crashes in the TLS lookup code, and why the core dump looks borked. 1) Apparently, Etch on ARM uses linuxthread instead of NPTL: what does $ getconf GNU_LIBPTHREAD_VERSION return on your box? 2) If it's using linxthreads, the culprit is likely the call to PyGILState_GetThisThreadState() from faulthandler_fatal_error(), which does a TLS lookup (which screws up because it's running in a user-allocated stack allocated with sigaltstack). However, this should only happen when a a fatal signal is handled by faulthandler, which should - AFAICT - only happen in test_faulthandler. Rebuilding faulthandler with #undef HAVE_SIGALTSTACK at the top of the file, should do the trick. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12936 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12881] ctypes: segfault with large structure field names
Charles-François Natali neolo...@free.fr added the comment: Looks good to me. -- nosy: +neologix stage: patch review - commit review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12881 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12936] armv5tejl: random segfaults in getaddrinfo()
Charles-François Natali neolo...@free.fr added the comment: 2) http://sources.redhat.com/bugzilla/show_bug.cgi?id=12453 We actually had another issue due to this particular libc bug: http://bugs.python.org/issue6059 Basically, the problem is that if some libraries are dynamically loaded in an interleaved way, the TLS can be returned uninitialized, hence the segfault upon access. This problem can show up now because the import orders for some modules have been modified: depending on the test that crashes - or rather the tests that run just before - you might be able to pinpoint it quickly (or you could maybe use ltrace -e dlopen). Apparently, Etch on ARM uses linuxthread instead of NPTL ... FYI you can also try to print sys.thread_info (which should give the same information, NPTL 2.7). NPTL has know issues: see for example the Python issue #4970. NPTL is old and has been replaced by pthread in the glibc on Linux. I think you're confusing with linuxthreads ;-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12936 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12936] armv5tejl: random segfaults in getaddrinfo()
Charles-François Natali neolo...@free.fr added the comment: Oh, and BTW, for the Backtrace stopped: frame did not save the PC, you might want to install the libc-dbg package. This might help in finding precisely where it's crashing. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12936 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12936] armv5tejl segfaults: sched_setaffinity() vs. pthread_setaffinity_np()
Charles-François Natali neolo...@free.fr added the comment: I think I got it: pthread_setaffinity_np() does not crash. Nice. Out of curiosity, I just looked at the source code, and it just does sched_setaffinity(thread-tid), so you can do the same with sched_setaffinity(syscall(SYS_gettid)) for the current thread. However, I don't think we should/could add this to the posix module: it expects a pthread_t instead of a PID, to which we don't have access. Furthermore, even though we're linked with pthread, this should normally succeed - or at least not crash - when called from the main thread - and it does on my Debian squeeze box. So I'd suggest closing this issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12936 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12936] armv5tejl segfaults: sched_setaffinity() vs. pthread_setaffinity_np()
Charles-François Natali neolo...@free.fr added the comment: If we have access (and as I understood from Victor's post we do): pthread_getaffinity_np() also exists on FreeBSD, which would be an advantage. Yes, but I see several drawbacks: - as noted by Victor, it's really easy to crash the interpreter by passing an invalid thread ID, which IMHO, should be avoided at all cost - to be safe, we would need to have a different API depending on whether Python is built with threads or not (i.e. sched_setaffinity() without threads, and pthread_setaffinity_np()) - pthread_setaffinity_np() is really non-portable (it's guarded by __USE_GNU in my system's header) - sched_setaffinity() seems to work fine on most systems even when linked with pthread I don't care strongly about using pthread_getaffinity_np(), but at least I'd like to skip the scheduling sections on arm-linux if they don't work reliably. Sounds reasonable. I guess you could use os.uname() or platform.machine(). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12936 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12936] armv5tejl segfaults: sched_setaffinity() vs. pthread_setaffinity_np()
Charles-François Natali neolo...@free.fr added the comment: Do you mean that signal.pthread_kill() should be removed? This function is very useful and solve some issues that cannot be solved differently. At the same time, I don't think that it's possible to workaround the crashes. At least, I don't see how: pthread_kill(tid, 0) is supposed to check if tid exists, but it does crash... No, I don't suggest to remove it, it is useful. As for the crashes, with glibc pthread_t is really a pointer, so there's no way to check its validity beforehand. Even if we did check the thread ID against the list of Python-created threads IDs (stored in Thread._ident), this could still crash, because the ID becomes invalid as soon as the thread terminates (all threads are started detached). Furthermore, this wouldn't work for non-Python created threads. We cannot use the same name for two different C function. One expects a process identifier, whereas the other expects a thread identifier! If Python is compiled without thread, the thread will not exist (as some modules and many other functions). I know, that's why I said different API: but I must admit it was poorly worded ;-) However, this wouldn't solve this particular problem: as long as we expose sched_setaffinity(), it will crash as soon as someone passes `0` or getpid() as PID. pthread_setaffinity_np() is really non-portable (it's guarded by __USE_GNU in my system's header) We can check it in configure. We already use some functions which are GNU extensions, like makedev(). Oh, os.makedev() availability is just not documented :-) As I said, this wouldn't solve this problem. If someone deems it necessary, we can open another issue for this feature request. sched_setaffinity() seems to work fine on most systems even when linked with pthread Again, it looks like a libc/kernel bug. I don't think that Python can work around such issue. Agreed. I don't know or need (), but the difference between sched_setaffinity and pthread_getaffinity_np is the same between sigprocmask() and pthread_sigmask(). I chose to expose only the later because the behaviour of sigprocmask is undefined in a process using threads. Exactly. However, nothing prevents someone from using sigprocmask() in a multithreaded process, the only difference is that it won't crash (AFAICT). So I suggest to: 1) skip the problematic tests on ARM when built with threads to avoid segfaults 2) if someone wants pthread_getaffinity_np(), then we can still open a separate feature request -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12936 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12975] Invitation to connect on LinkedIn
Changes by Charles-François Natali neolo...@free.fr: -- resolution: - invalid status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12975] Invitation to connect on LinkedIn
Changes by Charles-François Natali neolo...@free.fr: Removed file: http://bugs.python.org/file23150/unnamed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8828] Atomic function to rename a file
Charles-François Natali neolo...@free.fr added the comment: According to the following article, a fsync is also needed on the directory after a rename. I don't understand if is it always needed for an atomic rename, or if we only need it for the atomic write pattern. It's not needed if you just want atomicity, i.e. the file is visible either under its old name or its new name, but not neither or both. If is however needed if you want durability, i.e. you want to guarantee that the file is visible under its new name after your atomic_rename returns. -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8828 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12936] armv5tejl segfaults: sched_setaffinity() vs. pthread_setaffinity_np()
Charles-François Natali neolo...@free.fr added the comment: I'd prefer to disable the misbehaving functions entirely on arm. -10 If we start disabling features on platforms with partly bogus implementations, we might as well drop threading on OpenBSD, sendmsg() on OS-X, etc. Furthermore, it's really just a libc bug, which might be fixed in a more recent version, or with another libc provider (eglibc, uclibc, etc.). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12936 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12976] select module: only use EVFILT_TIMER if available (kqueue backend)
Charles-François Natali neolo...@free.fr added the comment: Hello, According to http://fxr.watson.org/fxr/ident?v=NETBSD;im=3;i=EVFILT_TIMER EVFILT_TIMER is defined on NetBSD. As for MirBSD, with all due respect, it really looks like a niche platform, definitely not officially supported by Python. Of course, this patch is so trivial and small that it can easily be merged, but it would be nice if MirBSD defined it in its header file instead (it's not the first problem due to kqueue-incompatibilities between on BSD platforms, see for example issue #12181 and issue #6419). -- nosy: +haypo, neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12976 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12976] select module: only use EVFILT_TIMER if available (kqueue backend)
Charles-François Natali neolo...@free.fr added the comment: Since this patch alone won't be enough to support MirBSD (and is required only for MirBSD), I suggest you to post the complete patch, and rename this issue add support for MirBSD platform, or something along those lines. That way, we can consider this as a feature request, and apply it in one chunk, if accepted. However, I think that the current consensus is somewhat hostile to adding support for so-called exotic platforms, so I can't guarantee this will be included (posting the whole patch will certainly help making a decision). I'm adding Martin to the noisy list, he'll probably give you more insight on this. -- nosy: +loewis ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12976 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
New submission from Charles-François Natali neolo...@free.fr: Now that sendmsg()/recvmsg() are exposed in socketmodule, we could use them to replace the ad-hoc FD-passing routines used by multiprocessing.reduction. Antoine suggested adding sendfd()/recvfd() methods to socket objects, but I'm not sure about this, since those only make sense for Unix domain sockets. Two remarks on the patch attached: - this removes sendfd()/recvfd() from _multiprocessing (but AFAICT those were never documented as part of the public API) - EOF/invalid data received result in a RuntimeError -- components: Library (Lib) files: multiprocessing_fd.diff keywords: patch messages: 144047 nosy: haypo, neologix priority: normal severity: normal status: open title: rewrite multiprocessing (senfd|recvfd) in Python type: feature request versions: Python 3.3 Added file: http://bugs.python.org/file23156/multiprocessing_fd.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Charles-François Natali neolo...@free.fr added the comment: I don't think that it's a problem to remove private functions. Alright. Is it mandatory to send a non-empty message (first argument for sendmsg, b'x' in your patch)? The original C function sends a random byte :-) Some implementation can return EINVAL if no data is sent (i.e. you can't send only ancillary data). multiprocessing_recvfd() contains cmsg_level=SOL_SOCKET and cmsg_type=SCM_RIGHTS, your Python function doesn't check cmsg_level or cmsg_type. Should it be checked? Yes, it should be checked, I'll update the patch. I don't know sendmsg/recvmsg API. Do they guarantee to send/receive all data? For data no, but ancillary data, yes. The only thing that could go wrong would be a buffer too short to hold the ancillary data, but: - the buffer size is computed with CMSG_DATA(), so it should be enough - if the ancillay data is truncated, struct.unpack will raise an exception -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Changes by Charles-François Natali neolo...@free.fr: Added file: http://bugs.python.org/file23166/multiprocessing_fd-1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Changes by Charles-François Natali neolo...@free.fr: Removed file: http://bugs.python.org/file23156/multiprocessing_fd.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Charles-François Natali neolo...@free.fr added the comment: I only tried on Linux. By the way, what's the simplest way to create a personal clone to test patches on some of the buildbots before committing them? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8237] multiprocessing.Queue() blocks program
Charles-François Natali neolo...@free.fr added the comment: It's a dupe of issue #8426: the Queue isn't full, but the underlying pipe is, so the feeder thread blocks on the write to the pipe (actually when trying to acquire the lock protecting the pipe from concurrent access). Since the children processes join the feeder thread on exit (to make sure all data has been flushed to pipe), they block. -- nosy: +neologix resolution: invalid - duplicate stage: test needed - committed/rejected status: open - closed superseder: - multiprocessing.Queue fails to get() very large objects ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8237 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12996] multiprocessing.Connection endianness issue
New submission from Charles-François Natali neolo...@free.fr: Since the rewrite in pure Python of multiprocessing.Connection (issue #11743), multiprocessing.Connection sends and receives the length of the data (used as header) in host byte order. This will break if the connection's endpoints are on machine with different endianness. Patch attached (it also removes an unnecessary computation of the length of the data being sent). -- components: Library (Lib) files: multiprocessing_conn_endianness.diff keywords: needs review, patch messages: 144148 nosy: haypo, neologix priority: normal severity: normal stage: patch review status: open title: multiprocessing.Connection endianness issue type: behavior versions: Python 3.3 Added file: http://bugs.python.org/file23171/multiprocessing_conn_endianness.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12996 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12999] _XOPEN_SOURCE usage on Solaris
Changes by Charles-François Natali neolo...@free.fr: -- nosy: haypo, neologix priority: normal severity: normal stage: needs patch status: open title: _XOPEN_SOURCE usage on Solaris type: behavior versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12999 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12999] _XOPEN_SOURCE usage on Solaris
New submission from Charles-François Natali neolo...@free.fr:
While testing issue #12981, I stumbled on a problem on OpenIndiana buildbot:
test test_multiprocessing crashed -- Traceback (most recent call last):
File
/export/home/buildbot/64bits/custom.cea-indiana-amd64/build/Lib/test/regrtest.py,
line 1133, in runtest_inner
the_package = __import__(abstest, globals(), locals(), [])
File
/export/home/buildbot/64bits/custom.cea-indiana-amd64/build/Lib/test/test_multiprocessing.py,
line 38, in module
from multiprocessing import util, reduction
File
/export/home/buildbot/64bits/custom.cea-indiana-amd64/build/Lib/importlib/_bootstrap.py,
line 437, in load_module
return self._load_module(fullname)
File
/export/home/buildbot/64bits/custom.cea-indiana-amd64/build/Lib/importlib/_bootstrap.py,
line 141, in decorated
return fxn(self, module, *args, **kwargs)
File
/export/home/buildbot/64bits/custom.cea-indiana-amd64/build/Lib/importlib/_bootstrap.py,
line 342, in _load_module
exec(code_object, module.__dict__)
File
/export/home/buildbot/64bits/custom.cea-indiana-amd64/build/Lib/multiprocessing/reduction.py,
line 57, in module
raise ImportError('pickling of connections not supported')
ImportError: pickling of connections not supported
Which means that socket.CMSG_LEN isn't defined.
Now, you might wonder how this can work in the C version of
multiprocessing.(sendfd|recvfd), which needs CMSG_LEN().
Here's how:
#ifdef __sun
/* The control message API is only available on Solaris
if XPG 4.2 or later is requested. */
#define _XOPEN_SOURCE 500
#endif
And indeed:
http://fxr.watson.org/fxr/source/common/sys/socket.h?v=OPENSOLARIS#L478
#if defined(_XPG4_2)
/*
* The cmsg headers (and macros dealing with them) were made available as
* part of UNIX95 and hence need to be protected with a _XPG4_2 define.
*/
The problem is that socketmodule uses pyconfig.h defines, and _XOPEN_SOURCE
isn't defined on Solaris:
http://hg.python.org/cpython/rev/7c947768b435
(it was added explicitely to Modules/_multiprocessing/multiprocessing.h for
sendmsg by http://hg.python.org/cpython/rev/419901e65dd2).
So, _XOPEN_SOURCE is needed on Solaris to build socket_sendmsg and friends.
I'm not sure about the best way to proceed, since Martin certainly had good
reasons to remove _XOPEN_SOURCE definition entirely on Solaris.
Should we define it only at the top of socketmodule?
--
nosy: +loewis
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12999
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Charles-François Natali neolo...@free.fr added the comment: Did you try it on Linux, FreeBSD and/or Windows? It works fine on Linux, FreeBSD, OS X and Windows, but not on Solaris: see issue #12999. -- dependencies: +_XOPEN_SOURCE usage on Solaris ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12976] add support for MirBSD platform
Charles-François Natali neolo...@free.fr added the comment: Hello Benny, As requested, here is the full patch for MirBSD support. The diff was taken against version 2.7.2. It is really quite easy, you just need to handle MirBSD like OpenBSD. With this patch, I can successfully compile and run Python on MirBSD. Even though it is a rather exotic platform, I encourage you to take these changes, as they are quite minimal. Indeed, it's quite short and manageable, but see http://bugs.python.org/issue11937, especially Martin's and Terry's comments: Guido established a policy a few years ago that we should rather not incorporate support for every minority platform for which people contribute patches. While I'd personally agree that an Interix port would certainly be fun, pragmatically, I'm -1 on having the code in the code basis, and propose to close this issue as won't fix. We would certainly be happy to link to gentoo prefix from the other ports page on python.org. and Markus, I agree with Martin that this patch would go against current policy and should be closed. Rather than close it myself, I will try to persuade you to do so. [...] This patch is much simpler and cleaner though (OTOH, it's so simple it shouldn't be too much work for MirBSD folks to keep this patch in sync). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12976 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Charles-François Natali neolo...@free.fr added the comment:
Here's a patch taking into account the fact that
multiprocessing.reduction might not be available and importing it can
raise an ImportError (which is already the case with the C
implementation, but multiprocessing.reduction tests have been added
recently to test_multiprocessing), e.g. if the OS doesn't support FD
passing.
With this patch, the pure Python version can be applied, and passes on
Linux, FreeBSD, OS X, Windows and OpenSolaris (except that it's not
available on OpenSolaris until issue #12999 gets fixed).
I also slightly modified the struct format used in the pure Python
version to make sure the length is sent as a a native int (@i)
instead of a standardized int (=i), which might break if sizeof(int)
!= 4 (not sure there are many ILP64 architectures out there, but you
never know...).
--
Added file: http://bugs.python.org/file23179/skip_reduction.diff
Added file: http://bugs.python.org/file23180/multiprocessing_fd-2.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12981
___diff -r c6d52971dd2a Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py Thu Sep 15 18:18:51 2011 +0200
+++ b/Lib/test/test_multiprocessing.py Sat Sep 17 10:54:10 2011 +0200
@@ -35,7 +35,13 @@
import multiprocessing.heap
import multiprocessing.pool
-from multiprocessing import util, reduction
+from multiprocessing import util
+
+try:
+from multiprocessing import reduction
+HAS_REDUCTION = True
+except ImportError:
+HAS_REDUCTION = False
try:
from multiprocessing.sharedctypes import Value, copy
@@ -1631,6 +1637,7 @@
os.write(fd, data)
os.close(fd)
+@unittest.skipUnless(HAS_REDUCTION, test needs multiprocessing.reduction)
def test_fd_transfer(self):
if self.TYPE != 'processes':
self.skipTest(only makes sense with processes)
@@ -1648,6 +1655,7 @@
with open(test.support.TESTFN, rb) as f:
self.assertEqual(f.read(), bfoo)
+@unittest.skipUnless(HAS_REDUCTION, test needs multiprocessing.reduction)
@unittest.skipIf(sys.platform == win32,
test semantics don't make sense on Windows)
@unittest.skipIf(MAXFD = 256,
@@ -1987,10 +1995,12 @@
'multiprocessing', 'multiprocessing.connection',
'multiprocessing.heap', 'multiprocessing.managers',
'multiprocessing.pool', 'multiprocessing.process',
-'multiprocessing.reduction',
'multiprocessing.synchronize', 'multiprocessing.util'
]
+if HAS_REDUCTION:
+modules.append('multiprocessing.reduction')
+
if c_int is not None:
# This module requires _ctypes
modules.append('multiprocessing.sharedctypes')
diff -r c6d52971dd2a Lib/multiprocessing/reduction.py
--- a/Lib/multiprocessing/reduction.py Thu Sep 15 18:18:51 2011 +0200
+++ b/Lib/multiprocessing/reduction.py Fri Sep 16 19:44:51 2011 +0200
@@ -39,6 +39,7 @@
import sys
import socket
import threading
+import struct
import _multiprocessing
from multiprocessing import current_process
@@ -51,7 +52,8 @@
#
#
-if not(sys.platform == 'win32' or hasattr(_multiprocessing, 'recvfd')):
+if not(sys.platform == 'win32' or (hasattr(socket, 'CMSG_LEN') and
+ hasattr(socket, 'SCM_RIGHTS'))):
raise ImportError('pickling of connections not supported')
#
@@ -77,10 +79,23 @@
else:
def send_handle(conn, handle, destination_pid):
-_multiprocessing.sendfd(conn.fileno(), handle)
+with socket.fromfd(conn.fileno(), socket.AF_UNIX, socket.SOCK_STREAM)
as s:
+s.sendmsg([b'x'], [(socket.SOL_SOCKET, socket.SCM_RIGHTS,
+struct.pack(@i, handle))])
def recv_handle(conn):
-return _multiprocessing.recvfd(conn.fileno())
+size = struct.calcsize(@i)
+with socket.fromfd(conn.fileno(), socket.AF_UNIX, socket.SOCK_STREAM)
as s:
+msg, ancdata, flags, addr = s.recvmsg(1, socket.CMSG_SPACE(size))
+try:
+cmsg_level, cmsg_type, cmsg_data = ancdata[0]
+if (cmsg_level == socket.SOL_SOCKET and
+cmsg_type == socket.SCM_RIGHTS):
+return struct.unpack(@i, cmsg_data[:size])[0]
+except (ValueError, IndexError, struct.error):
+pass
+raise RuntimeError('Invalid data received')
+
#
# Support for a per-process server thread which caches pickled handles
diff -r c6d52971dd2a Modules/_multiprocessing/multiprocessing.c
--- a/Modules/_multiprocessing/multiprocessing.cThu Sep 15 18:18:51
2011 +0200
+++ b/Modules/_multiprocessing/multiprocessing.cFri Sep 16 19:44:51
2011 +0200
@@ -8,11 +8,6 @@
#include multiprocessing.h
-#ifdef SCM_RIGHTS
-#define
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Changes by Charles-François Natali neolo...@free.fr: Removed file: http://bugs.python.org/file23166/multiprocessing_fd-1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13001] test_socket.testRecvmsgTrunc failure on FreeBSD 7.2 buildbot
New submission from Charles-François Natali neolo...@free.fr: http://www.python.org/dev/buildbot/all/builders/x86 FreeBSD 7.2 3.x/builds/2129/steps/test/logs/stdio == FAIL: testRecvmsgTrunc (test.test_socket.RecvmsgUDPTest) -- Traceback (most recent call last): File /usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_socket.py, line 1666, in testRecvmsgTrunc self.checkFlags(flags, eor=False) File /usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_socket.py, line 1354, in checkFlags self.assertEqual(flags mask, checkset mask) AssertionError: 0 != 16 == FAIL: testRecvmsgTrunc (test.test_socket.RecvmsgIntoUDPTest) -- Traceback (most recent call last): File /usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_socket.py, line 1666, in testRecvmsgTrunc self.checkFlags(flags, eor=False) File /usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_socket.py, line 1354, in checkFlags self.assertEqual(flags mask, checkset mask) AssertionError: 0 != 16 This fails because MSG_TRUNC isn't always set in msg_flags when receiving a truncated datagram with recvmsg(). It's a known kernel bug (http://svnweb.freebsd.org/base?view=revisionrevision=211030), fixed in FreeBSD 8 (and the test indeed passes on the FreeBSD 8 buildbot). The patch attached skips the test on FreeBSD 8 (and introduces @support.requires_freebsd_version). -- components: Tests files: freebsd_msgtrunc.diff keywords: needs review, patch messages: 144188 nosy: haypo, neologix priority: normal severity: normal stage: patch review status: open title: test_socket.testRecvmsgTrunc failure on FreeBSD 7.2 buildbot type: behavior versions: Python 3.3 Added file: http://bugs.python.org/file23182/freebsd_msgtrunc.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13001 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12996] multiprocessing.Connection endianness issue
Charles-François Natali neolo...@free.fr added the comment:
Since the rewrite in pure Python of multiprocessing.Connection (issue
#11743), multiprocessing.Connection sends and receives the length of the data
(used as header) in host byte order.
I don't think so, the C code uses also the host endian. This issue is a
feature request.
No.
http://hg.python.org/cpython/file/5deecc04b7a2/Modules/_multiprocessing/socket_connection.c
In conn_send_string():
/* The header of the message is a 32 bit unsigned number (in
network order) which specifies the length of the body. If
the message is shorter than about 16kb then it is quicker to
combine the header and the body of the message and send
them at once. */
[...]
*(UINT32*)message = htonl((UINT32)length);
in conn_recv_string():
ulength = ntohl(ulength);
I don't know if anyone uses multiprocessing on different hosts (because it
doesn't work currently).
If you would like to support using multiprocessing on different hosts, it
should be documented in multiprocessing doc.
It does work, it's even documented ;-)
http://docs.python.org/dev/library/multiprocessing.html#multiprocessing-managers
A manager object returned by Manager() controls a server process which
holds Python objects and allows other processes to manipulate them
using proxies.
[...]
Server process managers are more flexible than using shared memory
objects because they can be made to support arbitrary object types.
Also, a single manager can be shared by processes on different
computers over a network. They are, however, slower than using shared
memory.
Managers use multiprocessing.connection to serialize data and send
them over a socket:
http://hg.python.org/cpython/file/5deecc04b7a2/Lib/multiprocessing/managers.py
#
# Mapping from serializer name to Listener and Client types
#
listener_client = {
'pickle' : (connection.Listener, connection.Client),
'xmlrpclib' : (connection.XmlListener, connection.XmlClient)
}
Yeah, Python's awesome :-)
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12996
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Charles-François Natali neolo...@free.fr added the comment: It works fine on Linux, FreeBSD, OS X and Windows, but not on Solaris: see issue #12999. Oh, thank for testing before committing :) It's hard to debug multiprocessing. Yes. Especially when you stumble upon a kernel/libc bug 25% of the time... So, what should I do? Apply the test catching the multiprocessing.connection ImportError to test_multiprocessing (which is necessary even with the current C version)? And then apply the pure Python version, or wait until the OpenIndiana case gets fixed? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Changes by Charles-François Natali neolo...@free.fr: Removed file: http://bugs.python.org/file23180/multiprocessing_fd-2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Charles-François Natali neolo...@free.fr added the comment: I had a look at this patch, and the FD passing looked OK, except that calculating the buffer size with CMSG_SPACE() may allow more than one file descriptor to be received, with the extra one going unnoticed - it should use CMSG_LEN() instead Thanks for catching this. Here's an updated patch. (the existing C implementation has the same problem, I see). I just checked, and the C version uses CMSG_SPACE() as the buffer size, but passes CMSG_LEN() to cmsg-cmsg_len and msg.msg_controllen. Or am I missing something? -- Added file: http://bugs.python.org/file23189/multiprocessing_fd-3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12996] multiprocessing.Connection endianness issue
Changes by Charles-François Natali neolo...@free.fr: -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12996 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Charles-François Natali neolo...@free.fr added the comment: I committed the patch to catch the ImportError in test_multiprocessing. I'll commit the other patch (pure Python version) in a couple days. Ah, no, you're right - that's fine. Sorry for the false alarm. No problem. As they say, better safe than sorry. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13022] _multiprocessing.recvfd() doesn't check that file descriptor was actually received
Charles-François Natali neolo...@free.fr added the comment: The patch includes a test case, but like the other recently-added tests for the function, it isn't guarded against multiprocessing.reduction being unavailable. Issue #12981 has a patch skip_reduction.diff (already in 3.3) to fix this, I'll apply skip_reduction.diff and your patch to 2.7 and 3.2. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13022 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10141] SocketCan support
Charles-François Natali neolo...@free.fr added the comment: Here's an updated patch, with more tests. Please review! -- keywords: +needs review nosy: +haypo stage: patch review - commit review Added file: http://bugs.python.org/file23225/socketcan_v4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10141 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10141] SocketCan support
Charles-François Natali neolo...@free.fr added the comment:
- dummy question: why an address is a tuple with 1 string instead of just
the string? Does AF_UNIX also uses a tuple of 1 string?
I think the reason behind the tuple is future proofing.
Here's the definition of `struct sockaddr_can` in my Linux box's headers:
/**
* struct sockaddr_can - the sockaddr structure for CAN sockets
* @can_family: address family number AF_CAN.
* @can_ifindex: CAN network interface index.
* @can_addr:protocol specific address information
*/
struct sockaddr_can {
sa_family_t can_family;
int can_ifindex;
union {
/* transport protocol class address information (e.g. ISOTP) */
struct { canid_t rx_id, tx_id; } tp;
/* reserved for future CAN protocols address information */
} can_addr;
};
By making it a tuple, it will be easier to extend the address that
must be passed to bind(2), should it ever evolve, in a backward
compatible way. Well, that's just a guess (I'm by no means a SocketCAN
expert :-).
- the example should also use struct.pack() to create the frame, I don't
like hardcoded BLOB
Done.
- in test_socket: _have_socket_can() interprets permission denied as CAN
is not supported, it would be nice to provide a better skip message. Create
maybe a decorator based?
AFAICT, it shouldn't fail with EPERM or so.
Also, I'm not sure what the message would look like, and it's probably
a bit overkill.
- _have_socket_can(): you may move s.close() outside the try block (add
maybe a else: block?) because you may hide a real bug in .close()
Changed that.
- data += b'\0' * (8 - can_dlc): I prefer data = data.ljust(8, '\x00')
Hum... Done.
- you might add frame encoder/decoder in your example
Done.
- if (!strcmp(PyBytes_AS_STRING(interfaceName), )) hum.
PyBytes_GET_SIZE(intername)==0 should be enough
Done.
- you truncate the interface name, it can be surprising, I would prefer an
error (e.g. interface name too long: 20 characters, the maximum is 10
characters ?)
I changed that, and added a test. Also, note that AF_PACKET suffers
from the same problem.
I'll submit a separate patch.
- (oh no! don't include horrible configure diff in patches for the bug
tracker :-p)
Yeah, I usually take care of that, but forgot this time.
In which Linux version was CAN introduced?
Apparently, 2.6.25. Note that we don't need
@support.requires_linux_version() though, it should be catched by
HAVE_SOCKET_CAN (also, you can't use it as a class decorator...).
Here's the updated patch. It passes on all the buildbots (of course,
it's only relevant on Linux).
--
Added file: http://bugs.python.org/file23234/socketcan_v5.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10141
___diff -r a06ef7ab7321 Doc/library/socket.rst
--- a/Doc/library/socket.rstWed Sep 21 22:05:01 2011 +0200
+++ b/Doc/library/socket.rstFri Sep 23 23:27:19 2011 +0200
@@ -80,6 +80,11 @@
If *addr_type* is TIPC_ADDR_ID, then *v1* is the node, *v2* is the
reference, and *v3* should be set to 0.
+- A tuple ``(interface, )`` is used for the :const:`AF_CAN` address family,
+ where *interface* is a string representing a network interface name like
+ ``'can0'``. The network interface name ``''`` can be used to receive packets
+ from all network interfaces of this family.
+
- Certain other address families (:const:`AF_BLUETOOTH`, :const:`AF_PACKET`)
support specific representations.
@@ -216,6 +221,19 @@
in the Unix header files are defined; for a few symbols, default values are
provided.
+.. data:: AF_CAN
+ PF_CAN
+ SOL_CAN_*
+ CAN_*
+
+ Many constants of these forms, documented in the Linux documentation, are
+ also defined in the socket module.
+
+ Availability: Linux = 2.6.25.
+
+ .. versionadded:: 3.3
+
+
.. data:: SIO_*
RCVALL_*
@@ -387,10 +405,14 @@
Create a new socket using the given address family, socket type and protocol
number. The address family should be :const:`AF_INET` (the default),
- :const:`AF_INET6` or :const:`AF_UNIX`. The socket type should be
- :const:`SOCK_STREAM` (the default), :const:`SOCK_DGRAM` or perhaps one of
the
- other ``SOCK_`` constants. The protocol number is usually zero and may be
- omitted in that case.
+ :const:`AF_INET6`, :const:`AF_UNIX` or :const:`AF_CAN`. The socket type
+ should be :const:`SOCK_STREAM` (the default), :const:`SOCK_DGRAM`,
+ :const:`SOCK_RAW` or perhaps one of the other ``SOCK_`` constants. The
+ protocol number is usually zero and may be omitted in that case or
+ :const:`CAN_RAW` in case the address family is :const:`AF_CAN`.
+
+ .. versionchanged:: 3.3
+ The AF_CAN family was added.
.. function:: socketpair([family[, type[, proto]]])
@@ -1213,7 +1235,7 @@
print('Received', repr(data
[issue10141] SocketCan support
Changes by Charles-François Natali neolo...@free.fr: Removed file: http://bugs.python.org/file23225/socketcan_v4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10141 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Changes by Charles-François Natali neolo...@free.fr: -- dependencies: -_XOPEN_SOURCE and _XOPEN_SOURCE_EXTENDED usage on Solaris resolution: - fixed stage: - committed/rejected ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12981] rewrite multiprocessing (senfd|recvfd) in Python
Changes by Charles-François Natali neolo...@free.fr: -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12981 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13058] Fix file descriptor leak on error
Charles-François Natali neolo...@free.fr added the comment: Patch applied, thanks! -- nosy: +neologix resolution: - fixed stage: - committed/rejected status: open - closed versions: +Python 2.7, Python 3.2, Python 3.3 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13058 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13070] segmentation fault in pure-python multi-threaded server
Charles-François Natali neolo...@free.fr added the comment:
Confirmed with default.
The problem is that the TextIOWrapper gets collected after the underlying
BufferedRWPair has been cleared (tp_clear) by the garbage collector: when
_PyIOBase_finalize() is called for the TextIOWrapper, it checks if the textio
is closed, which indirectly checks if the underlying rwpair is closed:
static PyObject *
bufferedrwpair_closed_get(rwpair *self, void *context)
{
return PyObject_GetAttr((PyObject *) self-writer, _PyIO_str_closed);
}
Since self-writer has already been set to NULL by bufferedrwpair_clear(),
PyObject_GetAttr() segfaults.
@Victor
Could you try the patch attached?
--
keywords: +patch
nosy: +amaury.forgeotdarc, neologix, pitrou
versions: +Python 3.3
Added file: http://bugs.python.org/file23277/buffered_closed_gc.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13070
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13070] segmentation fault in pure-python multi-threaded server
Charles-François Natali neolo...@free.fr added the comment: With test. -- Added file: http://bugs.python.org/file23278/buffered_closed_gc-1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13070 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13070] segmentation fault in pure-python multi-threaded server
Changes by Charles-François Natali neolo...@free.fr: Removed file: http://bugs.python.org/file23277/buffered_closed_gc.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13070 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13070] segmentation fault in pure-python multi-threaded server
Changes by Charles-François Natali neolo...@free.fr: Removed file: http://bugs.python.org/file23278/buffered_closed_gc-1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13070 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13070] segmentation fault in pure-python multi-threaded server
Changes by Charles-François Natali neolo...@free.fr: Added file: http://bugs.python.org/file23279/buffered_closed_gc-1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13070 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13084] test_signal failure
Charles-François Natali neolo...@free.fr added the comment: See http://bugs.python.org/issue12469, specifically http://bugs.python.org/issue12469#msg139831 When signals are unblocked, pending signal ared delivered in the reverse order of their number (also on Linux, not only on FreeBSD 6). I don't like this. POSIX doesn't make any guarantee about signal delivery order, except for real-time signals. It might work on FreeBSD and Linux, but that's definitely not documented, and might break with new kernel releases, or other kernels. It looks like it works like this on most OSes (Linux, Mac OS X, Solaris, FreeBSD): I don't see any test_signal failure on 3.x buildbots. If we have a failure, we can use set() again, but only for test_pending: signal order should be reliable if signals are not blocked. Looks like we now have a failure :-) -- nosy: +haypo, neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13084 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13084] test_signal failure
Changes by Charles-François Natali neolo...@free.fr: -- keywords: +patch Added file: http://bugs.python.org/file23284/check_signum_order.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13084 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13070] segmentation fault in pure-python multi-threaded server
Charles-François Natali neolo...@free.fr added the comment:
Shouldn't the test use self.BufferedRWPair instead of
io.BufferedRWPair?
Yes.
Also, is it ok to just return NULL or should the error state also be
set?
Well, I'm not sure, that why I made you and Amaury noisy :-)
AFAICT, this is the only case where _check_closed can encounter a NULL
self-writer. And this specific situation is not an error (nothing prevents the
rwpair from being garbaged collected before the textio) ,and
_PyIOBase_finalize() explicitely clears any error returned:
/* If `closed` doesn't exist or can't be evaluated as bool, then the
object is probably in an unusable state, so ignore. */
res = PyObject_GetAttr(self, _PyIO_str_closed);
if (res == NULL)
PyErr_Clear();
else {
closed = PyObject_IsTrue(res);
Py_DECREF(res);
if (closed == -1)
PyErr_Clear();
}
Furthermore, I'm not sure about what kind of error would make sense here.
--
Added file: http://bugs.python.org/file23285/buffered_closed_gc-2.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13070
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13070] segmentation fault in pure-python multi-threaded server
Changes by Charles-François Natali neolo...@free.fr: Removed file: http://bugs.python.org/file23279/buffered_closed_gc-1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13070 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13001] test_socket.testRecvmsgTrunc failure on FreeBSD 7.2 buildbot
Charles-François Natali neolo...@free.fr added the comment:
@requires_freebsd_version should be factorized with
@requires_linux_version.
Patches attached.
Can we workaround FreeBSD ( 8) bug in C/Python?
Not really.
Or should we remove the function on FreeBSD 8?
There's really no reason to do that (and it's really a minor bug).
--
Added file: http://bugs.python.org/file23296/freebsd_msgtrunc-1.diff
Added file: http://bugs.python.org/file23297/requires_unix_version.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13001
___diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -1659,6 +1659,9 @@
def _testRecvmsgShorter(self):
self.sendToServer(MSG)
+# FreeBSD 8 doesn't always set the MSG_TRUNC flag when a truncated
+# datagram is received (issue #13001).
+@support.requires_freebsd_version(8)
def testRecvmsgTrunc(self):
# Receive part of message, check for truncation indicators.
msg, ancdata, flags, addr = self.doRecvmsg(self.serv_sock,
@@ -1668,6 +1671,7 @@
self.assertEqual(ancdata, [])
self.checkFlags(flags, eor=False)
+@support.requires_freebsd_version(8)
def _testRecvmsgTrunc(self):
self.sendToServer(MSG)
diff --git a/Lib/test/support.py b/Lib/test/support.py
--- a/Lib/test/support.py
+++ b/Lib/test/support.py
@@ -44,8 +44,8 @@
Error, TestFailed, ResourceDenied, import_module,
verbose, use_resources, max_memuse, record_original_stdout,
get_original_stdout, unload, unlink, rmtree, forget,
-is_resource_enabled, requires, requires_linux_version,
-requires_mac_ver, find_unused_port, bind_port,
+is_resource_enabled, requires, requires_freebsd_version,
+requires_linux_version, requires_mac_ver, find_unused_port,
bind_port,
IPV6_ENABLED, is_jython, TESTFN, HOST, SAVEDCWD, temp_cwd,
findfile, create_empty_file, sortdict, check_syntax_error,
open_urlresource,
check_warnings, CleanImport, EnvironmentVarGuard,
TransientResource,
@@ -312,17 +312,17 @@
msg = Use of the %r resource not enabled % resource
raise ResourceDenied(msg)
-def requires_linux_version(*min_version):
-Decorator raising SkipTest if the OS is Linux and the kernel version is
-less than min_version.
+def _requires_unix_version(sysname, min_version):
+Decorator raising SkipTest if the OS is `sysname` and the version is
less
+than `min_version`.
-For example, @requires_linux_version(2, 6, 35) raises SkipTest if the Linux
-kernel version is less than 2.6.35.
+For example, @_requires_unix_version('FreeBSD', (7, 2)) raises SkipTest if
+the FreeBSD version is less than 7.2.
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kw):
-if sys.platform == 'linux':
+if platform.system() == sysname:
version_txt = platform.release().split('-', 1)[0]
try:
version = tuple(map(int, version_txt.split('.')))
@@ -332,13 +332,29 @@
if version min_version:
min_version_txt = '.'.join(map(str, min_version))
raise unittest.SkipTest(
-Linux kernel %s or higher required, not %s
-% (min_version_txt, version_txt))
-return func(*args, **kw)
-wrapper.min_version = min_version
+%s version %s or higher required, not %s
+% (sysname, min_version_txt, version_txt))
return wrapper
return decorator
+def requires_freebsd_version(*min_version):
+Decorator raising SkipTest if the OS is FreeBSD and the FreeBSD version
is
+less than `min_version`.
+
+For example, @requires_freebsd_version(7, 2) raises SkipTest if the FreeBSD
+version is less than 7.2.
+
+return _requires_unix_version('FreeBSD', min_version)
+
+def requires_linux_version(*min_version):
+Decorator raising SkipTest if the OS is Linux and the Linux version is
+less than `min_version`.
+
+For example, @requires_linux_version(2, 6, 32) raises SkipTest if the Linux
+version is less than 2.6.32.
+
+return _requires_unix_version('Linux', min_version)
+
def requires_mac_ver(*min_version):
Decorator raising SkipTest if the OS is Mac OS X and the OS X
version if less than min_version.
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10141] SocketCan support
Charles-François Natali neolo...@free.fr added the comment: So, Victor, what do you think of the last version? This patch has been lingering for quite some time, and it's really a cool feature. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10141 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13084] test_signal failure
Changes by Charles-François Natali neolo...@free.fr: -- resolution: - fixed stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13084 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13045] socket.getsockopt may require custom buffer contents
Charles-François Natali neolo...@free.fr added the comment: Hello, method:: socket.getsockopt(level, optname[, optarg]) The overloading of the third parameter is confusing: it can already be an integer value or a buffer size, I don't think that adding a third possibility is a good idea. It might be better to add another optional `buffer` argument (and ignore `buflen` if this argument is provided). Also, it would be nice to have a corresponding unit test: since I doubt this buffer argument is supported by many Unices out there, you can probably reuse a subset of what ipset does (just take care and guard it by @support.requires_linux_version() if applicable). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13045 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13001] test_socket.testRecvmsgTrunc failure on FreeBSD 7.2 buildbot
Changes by Charles-François Natali neolo...@free.fr: -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13001 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12156] test_multiprocessing.test_notify_all() timeout (1 hour) on FreeBSD 7.2
Charles-François Natali neolo...@free.fr added the comment: test_multiprocessing frequently hangs on FreeBSD 8 buildbots, and this probably has to do with the limit on the max number of POSIX semaphores: == ERROR: test_notify_all (test.test_multiprocessing.WithProcessesTestCondition) -- Traceback (most recent call last): File /usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py, line 777, in test_notify_all cond = self.Condition() File /usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/__init__.py, line 189, in Condition return Condition(lock) File /usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/synchronize.py, line 198, in __init__ self._lock = lock or RLock() File /usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/synchronize.py, line 172, in __init__ SemLock.__init__(self, RECURSIVE_MUTEX, 1, 1) File /usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/synchronize.py, line 75, in __init__ sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue) OSError: [Errno 23] Too many open files in system There are probably dangling semaphores, since the test doesn't use that much POSIX semaphores. Either way, we can't do much about it... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12156 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13070] segmentation fault in pure-python multi-threaded server
Charles-François Natali neolo...@free.fr added the comment: Probably. OTOH, not setting the error state when returning NULL is usually an error (and can result in difficult-to-debug problems), so let's stay on the safe side. RuntimeError perhaps. OK, I'll update the patch accordingly. Does that mean that an application will see a Python exception? No, the finalization code explicitly clears any exception set. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13070 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10348] multiprocessing: use SysV semaphores on FreeBSD
Charles-François Natali neolo...@free.fr added the comment: -1 IMHO, implementing SysV semaphores would be a step backwards, plus the API is a real pain. I think there's no reason to complicate the code to accomodate such corner cases, especially since the systems that don't support POSIX semaphores will eventually die out... -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10348 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13070] segmentation fault in pure-python multi-threaded server
Charles-François Natali neolo...@free.fr added the comment:
Sorry, forgot about this issue...
Updated patch (I'm not really satisfied with the error message, don't
hesitate if you can think of a better wording).
--
Added file: http://bugs.python.org/file23319/buffered_closed_gc-3.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13070
___diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -2421,6 +2421,20 @@
with self.open(support.TESTFN, rb) as f:
self.assertEqual(f.read(), b456def)
+def test_rwpair_cleared_before_textio(self):
+# Issue 13070: TextIOWrapper's finalization would crash when called
+# after the reference to the underlying BufferedRWPair got cleared.
+for i in range(1000):
+b1 = self.BufferedRWPair(self.MockRawIO(), self.MockRawIO())
+t1 = self.TextIOWrapper(b1, encoding=ascii)
+b2 = self.BufferedRWPair(self.MockRawIO(), self.MockRawIO())
+t2 = self.TextIOWrapper(b2, encoding=ascii)
+# circular references
+t1.buddy = t2
+t2.buddy = t1
+support.gc_collect()
+
+
class PyTextIOWrapperTest(TextIOWrapperTest):
pass
diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c
--- a/Modules/_io/bufferedio.c
+++ b/Modules/_io/bufferedio.c
@@ -2307,6 +2307,10 @@
static PyObject *
bufferedrwpair_closed_get(rwpair *self, void *context)
{
+if (self-writer == NULL) {
+PyErr_SetString(PyExc_RuntimeError, the writer object has been
cleared);
+return NULL;
+}
return PyObject_GetAttr((PyObject *) self-writer, _PyIO_str_closed);
}
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13045] socket.getsockopt may require custom buffer contents
Charles-François Natali neolo...@free.fr added the comment: I've attached an update for the previous patch. Now there's no more overloading for the third argument and socket.getsockopt accepts one more optional argument -- a buffer to use as an input to kernel. Remarks: + length. If *buffer* is absent and *buflen* is an integer, then *buflen* [...] + this buffer is returned as a bytes object. If *buflen* is absent, an integer There's a problem here, the first buflen part should probably be removed. Also, you might want to specify that if a custom buffer is provided, the length argument will be ignored. By the way, I don't really think that any POSIX-compliant UNIX out there would treat the buffer given to getsockopt in any way different from what Linux does. It is very easy to copy the buffer from user to kernel and back, and it is so inconvenient to prevent kernel from reading it prior to modification, that I bet no one has ever bothered to do this. Me neither, I don't expect the syscall to return EINVAL: the goal is just to test the correct passing of the input buffer, and the length computation. If we can't test this easily within test_socket, it's ok, I guess the following should be enough: - try supplying a non-buffer argument as fourth parameter (e.g. and int), and check that you get a ValueError - supply a buffer with a size == sizeof(int) (SIZEOF_INT is defined in Lib/test/test_socket.py), and call getsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 0, buffer): this should normally succeed, and return a buffer (check the return type) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13045 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10141] SocketCan support
Changes by Charles-François Natali neolo...@free.fr: -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10141 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11956] 3.3 : test_import.py causes 'make test' to fail
Changes by Charles-François Natali neolo...@free.fr: -- resolution: - fixed stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11956 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10141] SocketCan support
Charles-François Natali neolo...@free.fr added the comment: I don't have much to say about the patch, given that I don't know anything about CAN and my system doesn't appear to have a vcan0 interface. I had never heard about it before this issue, but the protocol is really simple. If you want to try it out (just for fun :-), you just have to do the following: # modprobe vcan # ip link add dev vcan0 type vcan # ifconfig vcan0 up -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10141 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13070] segmentation fault in pure-python multi-threaded server
Charles-François Natali neolo...@free.fr added the comment: Committed to 3.2 and default. Victor, thanks for the report! -- resolution: - fixed stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13070 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13070] segmentation fault in pure-python multi-threaded server
Charles-François Natali neolo...@free.fr added the comment: The issue doesn't affect Python 2.7? Duh! I was sure the _io module had been introduced in Python 3 (I/O layer rewrite, etc). Yes, it does apply to 2.7. I'll commit the patch later today. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13070 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7719] distutils: ignore .nfsXXXX files
Charles-François Natali neolo...@free.fr added the comment: Wouldn't it be better to add an 'ignore' option to the copy_tree() method with an optional list of patterns to ignore instead of hardcoding '.nsfXXX' files? This would make it possible to also skip '.hg', 'CVS' artifacts. -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7719 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14702] os.makedirs breaks under autofs directories
Charles-François Natali neolo...@free.fr added the comment: I guess this is the magic in mkdir -p: mkdir(expert, 0755) = -1 EACCES (Permission denied) chdir(expert) = 0 mkdir(tmp, 0755) = -1 EEXIST (File exists) I'm not sure how I feel about that. I think we have more or less a policy in os shutil not to change directories unless really necessary (like make_archive). Calling chdir() should be avoided, as it is not thread safe. Actually, the real problem is that stat() alone doesn't trigger automount, to avoid mount storms e.g. when using find, see: https://bugzilla.redhat.com/show_bug.cgi?id=497830 http://lkml.indiana.edu/hypermail/linux/kernel/1109.1/00100.html Since calling chdir() is not an option, if we want to fix this, I guess the way to go would be to open() the directory, and not use path.exists(). But I find really uncomfortable with adding such a kludge, especially since this problem is probably present in many different places throughout the stdlib. Furthermore, it seems that kernel from 2.6.38 do trigger automount upon stat() (see http://lkml.indiana.edu/hypermail/linux/kernel/1109.1/00210.html), so I'd suggest to drop this as won't fix. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14702 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14702] os.makedirs breaks under autofs directories
Charles-François Natali neolo...@free.fr added the comment: Yes, creating the directories in a bottom-up way (i.e. '/net', '/net/prodigy', '/net/prodigy/foo') could maybe avoid this problem. But this is definietely an autofs bug, and there are probably many other places where such code might break. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14702 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14702] os.makedirs breaks under autofs directories
Charles-François Natali neolo...@free.fr added the comment: Alright, closing for good then. Andrew, if you want to get this fixed, you should report this to the autofs folks, because it's definitely not a Python bug. -- stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14702 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
