Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Le jeudi 30 décembre 2010 à 17:05 +0100, Martin v. Löwis a écrit : I really don't think it is our job to maintain a list of OS/versions which work and don't work. Of course not. I would propose a dynamic test: check how many POSIX semaphores the installation supports, and fail if it's less than 200 (say). I added informations about FreeBSD, NetBSD, Darwin and OpenBSD to the issue #10348: http://bugs.python.org/issue10348#msg125042 The maximum number of POSIX semaphores can be read with sysctl: - FreeBSD: p1003_1b.sem_nsems_max - NetBSD: kern.posix.semmax - Darwin: kern.posix.sem.max - OpenBSD: (no support) Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
I added informations about FreeBSD, NetBSD, Darwin and OpenBSD to the issue #10348: http://bugs.python.org/issue10348#msg125042 The maximum number of POSIX semaphores can be read with sysctl: - FreeBSD: p1003_1b.sem_nsems_max - NetBSD: kern.posix.semmax - Darwin: kern.posix.sem.max - OpenBSD: (no support) I've been using os.sysconf(SC_SEM_NSEMS_MAX), which seems to have worked fine (it's also what POSIX mandates). See #10798. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 30.12.2010 04:45, schrieb Brian Quinlan: On Dec 29, 2010, at 2:55 PM, Victor Stinner wrote: Le mercredi 29 décembre 2010 à 21:49 +0100, Martin v. Löwis a écrit : Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. More numbers (on Linux): - Queue: 3 SemLock - Condition: 4 SemLock - Event: 5 SemLock - Call (test_concurrent_futures): 10 SemLock (2 Event) - ProcessPoolExecutor: 11 SemLock (2 Queue, 1 Condition) FreeBSD 7.2 is limited to 30 semaphores, so with ProcessPoolExecutor, you can only create *one* Call object, whereas some tests use 4 Call objects or more. Great detective work! This would suggest that ProcessPoolExecutors are useable on FreeBSD 7.2 so long as the user doesn't create more than two at once (which probably isn't a big deal for most apps). IIUC, this is a system-wide limit. So if you run any non-trivial app a number of times will quickly cause failure. This is just not practical. So skipping the test is probably the way to go. I'm still -1 on that proposal. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Wiadomość napisana przez Martin v. Löwis w dniu 2010-12-30, o godz. 10:16: Am 30.12.2010 04:45, schrieb Brian Quinlan: So skipping the test is probably the way to go. I'm still -1 on that proposal. I agree with Martin, explanation follows. In general, I'm trying to think as the user: someone wrote a program and said it requires Python 3.2. I expect it to work on Python 3.2 available on my platform. Doesn't matter if it's bundled with the OS or custom built. When a module is not available, this should fail loudly and as fast as possible. This is currently the case with conditionally built modules like curses, sqlite3, tkinter and others. From the user's perspective: the import fails. From the administrator's perspective: the build reports skipped modules. The admin can then alter the environment to make the build behave as expected, and then run the tests to confirm it works. From an OS maintainer perspective it's even better: a module can be made available optionally with explicitly set dependencies and required configuration. In our specific case a bunch of things should be determined first: 1. Does it still fail on FreeBSD 7.3+? 2. Why is the semaphore limit so low in the first place? 3. What is a reasonable number for that limit? That should include situations where people run multiple applications. 4. What other configuration should be done to ensure the module won't break on runtime. Some answers Philip gave already. Knowing all these things would let us decide whether switching the module off on systems that don't meet the requirements is okay and can we get away with just documenting how to make it work. One problem is that the administrator can make the OS compatible/incompatible on runtime (with sysctl). ISTM concurrent.futures was included a bit too early in the distribution. The module has problems not only on BSD [1]. Fortunately, there is visible action from Brian, Philip and David to make the problems go away. I'm sure it will be fixed pretty soon. Ultimately I also agree with Martin that an implementation using SysV IPC would probably be better from the user's perspective. As he said, even if the underlying code is not as pretty as with the POSIX APIs, once it's written and tested nobody cares because it's not exposed. However, it's far too late to change that now for 3.2. [1] On RedHat/CentOS the test freezes badly (issue 10517). -- Best regards, Łukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Thu, 30 Dec 2010 16:04:16 +0100 Łukasz Langa luk...@langa.pl wrote: Some answers Philip gave already. Knowing all these things would let us decide whether switching the module off on systems that don't meet the requirements is okay and can we get away with just documenting how to make it work. I really don't think it is our job to maintain a list of OS/versions which work and don't work. At best, multiprocessing when imported could try to create a single semaphore, and raise an ImportError if such creating fails (which it does, IIUC, under some old FreeBSDs without a specific kernel tweak). ISTM concurrent.futures was included a bit too early in the distribution. The module has problems not only on BSD [1]. I'm not sure concurrent.futures is the culprit, rather than multiprocessing itself (or perhaps even some core Python functionality). Actually, the threading-based part of concurrent.futures probably works fine. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 30.12.2010 16:40, schrieb Antoine Pitrou: On Thu, 30 Dec 2010 16:04:16 +0100 Łukasz Langa luk...@langa.pl wrote: Some answers Philip gave already. Knowing all these things would let us decide whether switching the module off on systems that don't meet the requirements is okay and can we get away with just documenting how to make it work. I really don't think it is our job to maintain a list of OS/versions which work and don't work. Of course not. I would propose a dynamic test: check how many POSIX semaphores the installation supports, and fail if it's less than 200 (say). ISTM concurrent.futures was included a bit too early in the distribution. The module has problems not only on BSD [1]. I'm not sure concurrent.futures is the culprit, rather than multiprocessing itself (or perhaps even some core Python functionality). Actually, the threading-based part of concurrent.futures probably works fine. Well, the culprit really is FreeBSD. However, concurrent.futures apparently makes freehanded use of semaphores, in a way that the FreeBSD authors didn't expect them to be used (if they expected them to be used at all, that is). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
I'm not sure concurrent.futures is the culprit, rather than multiprocessing itself (or perhaps even some core Python functionality). Actually, the threading-based part of concurrent.futures probably works fine. Well, the culprit really is FreeBSD. However, concurrent.futures apparently makes freehanded use of semaphores, in a way that the FreeBSD authors didn't expect them to be used (if they expected them to be used at all, that is). It seems to be multiprocessing as much as concurrent.futures itself. Apparently a basic multiprocessing queue already uses two (interprocess) locks and one semaphore. And, again, the threading-based API in concurrent.futures should work fine anyway. Do you suggest we selectively disable the process-based API? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
And, again, the threading-based API in concurrent.futures should work fine anyway. Do you suggest we selectively disable the process-based API? Yes. Importing concurrent.futures.process should fail. Unfortunately, it's imported from __init__.py, so either we change the API to move the executors to the submodules, or we let creation of process pools fail (rather than the import). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
1. Does it still fail on FreeBSD 7.3+? Yes, it still fails. The limits (30 semaphores) haven't changed. It also remains untunable. 2. Why is the semaphore limit so low in the first place? I don't know - (Free)BSD is in the tradition of disliking SysV inventions, and POSIX inventions unless they originate from FreeBSD. This is polemics, of course :-) In any case, SysV IPC doesn't seem much better. In SysV, you allocate semaphores in sets. In FreeBSD 8.1, in the standard setting, you can have up to 10 semaphore identifiers, with up to 60 semaphores each. That may sound reasonable, except that there is also a limit on the total number of semaphores of 60. So it seems that switching to SysV IPC wouldn't actually improve things (except that these are tunable parameters already, so changing them requires only a reboot, not a recompile of the kernel). 3. What is a reasonable number for that limit? That should include situations where people run multiple applications. POSIX specifies that the minimum acceptable value for the SEM_NSEMS_MAX setting is _POSIX_SEM_NSEMS_MAX, which in turn must be at least 256. We could argue that we refuse to use POSIX semaphores if the system doesn't conform to POSIX, i.e. has semaphore limit of less than 256. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Martin v. Löwis wrote: And, again, the threading-based API in concurrent.futures should work fine anyway. Do you suggest we selectively disable the process-based API? Yes. Importing concurrent.futures.process should fail. If I understand correctly, it is possible to adjust BSD so that this will work -- as a user I would much rather be informed of that, even as just a caution, than to have Python not have the functionality even after I had fixed the OS problems. Mind you, I have no idea how hard it would be to make this happen in the module in an intelligent way. ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Thu, 30 Dec 2010 12:36:47 -0800 Ethan Furman et...@stoneleaf.us wrote: Martin v. Löwis wrote: And, again, the threading-based API in concurrent.futures should work fine anyway. Do you suggest we selectively disable the process-based API? Yes. Importing concurrent.futures.process should fail. If I understand correctly, it is possible to adjust BSD so that this will work -- as a user I would much rather be informed of that, even as just a caution, than to have Python not have the functionality even after I had fixed the OS problems. Mind you, I have no idea how hard it would be to make this happen in the module in an intelligent way. As I said, we can just emit a Warning instead of raising an exception. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 30.12.2010 19:17, schrieb Martin v. Löwis: 1. Does it still fail on FreeBSD 7.3+? Yes, it still fails. The limits (30 semaphores) haven't changed. It also remains untunable. 2. Why is the semaphore limit so low in the first place? I don't know - (Free)BSD is in the tradition of disliking SysV inventions, and POSIX inventions unless they originate from FreeBSD. This is polemics, of course :-) BTW - can anyone contribute data points from other *BSDs? Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 30.12.2010 21:36, schrieb Ethan Furman: Martin v. Löwis wrote: And, again, the threading-based API in concurrent.futures should work fine anyway. Do you suggest we selectively disable the process-based API? Yes. Importing concurrent.futures.process should fail. If I understand correctly, it is possible to adjust BSD so that this will work -- as a user I would much rather be informed of that, even as just a caution, than to have Python not have the functionality even after I had fixed the OS problems. That doesn't exclude each other. The module will fail on default (misconfigured) systems, and work on correctly-configured systems. See my patch for details. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
BTW - can anyone contribute data points from other *BSDs? I don't have an installation of OpenBSD, but... In FreeBSD, POSIX semaphores are implemented in sys/kern/uipc_sem.c. In http://www.openbsd.org/cgi-bin/cvsweb/src/sys/kern/ that file doesn't exist. Also, in FreeBSD's limits.h, _POSIX_SEM_NSEMS_MAX is defined (surprisingly to 256); in http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/include/limits.h?rev=1.15;content-type=text/plain this constant doesn't appear. So ISTM that OpenBSD doesn't implement POSIX semaphores. IIUC, this means that the multiprocessing module won't be fully functional, and its tests (and the concurrent.futures tests) will be skipped. NetBSD apparently supports sem_open since 2.0: http://netbsd.gw.com/cgi-bin/man-cgi?sem_open++NetBSD-current According to http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/kern/uipc_sem.c?rev=1.22content-type=text/x-cvsweb-markuponly_with_tag=MAIN SEM_MAX is 128 since 2007, and dynamically adjustable (no reboot). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Martin v. Löwis mar...@v.loewis.de writes: 1. Does it still fail on FreeBSD 7.3+? Yes, it still fails. The limits (30 semaphores) haven't changed. It also remains untunable. Yeah, my recollection about 7.3 appears to have been remembering when the kernel module was included by default as opposed to needing to be explicitly loaded. From the older discussion (http://mail.python.org/pipermail/python-dev/2010-November/105380.html) it would appear that 7.x remains fixed sans a kernel build (not necessarily a big deal for FreeBSD users), but 8.1+ can be tuned with sysctl. -- David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Hi, FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of a concurrent.futures failure. The problem is that test_concurrent_futures uses many (multiprocessing) POSIX semaphores, whereas POSIX semaphores support in FreeBSD is recent and limited. We have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: * http://bugs.python.org/issue10348 * Too many open files errors on x86 FreeBSD 7.2 3.x buildbot ^-- thread in python-dev opened last month I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Does anyone know SysV API? I tried to write a patch, but I never used semaphores (POSIX or SysV). There is a third party module which looks complete and stable: http://semanchuk.com/philip/sysv_ipc/ It is released under the BSD license. It supports semaphores, but also shared memory and message queues. We don't need all of those, semaphores would be enough. I added its author (Philip Semanchuk) to this thread. I don't know how we should decide to use POSIX or SysV semaphores. It looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD based OSes), and POSIX is preferred on Linux. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 29.12.2010 14:17, schrieb Victor Stinner: Hi, FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of a concurrent.futures failure. The problem is that test_concurrent_futures uses many (multiprocessing) POSIX semaphores, whereas POSIX semaphores support in FreeBSD is recent and limited. We have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: * http://bugs.python.org/issue10348 * Too many open files errors on x86 FreeBSD 7.2 3.x buildbot ^-- thread in python-dev opened last month I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Under the condition that it is within reason to fix it before the release. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 8:17 AM, Victor Stinner wrote: Hi, FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of a concurrent.futures failure. The problem is that test_concurrent_futures uses many (multiprocessing) POSIX semaphores, whereas POSIX semaphores support in FreeBSD is recent and limited. We have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: * http://bugs.python.org/issue10348 * Too many open files errors on x86 FreeBSD 7.2 3.x buildbot ^-- thread in python-dev opened last month I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Does anyone know SysV API? I tried to write a patch, but I never used semaphores (POSIX or SysV). There is a third party module which looks complete and stable: http://semanchuk.com/philip/sysv_ipc/ It is released under the BSD license. It supports semaphores, but also shared memory and message queues. We don't need all of those, semaphores would be enough. I added its author (Philip Semanchuk) to this thread. Hi all, What Victor says above is correct, although I wasn't aware that POSIX IPC under FreeBSD 7.2 was still having problems. Prior to 7.2 it was broken but 7.2 worked OK in my limited testing. In any case, the sysv_ipc module is mine and it's mature and you're welcome to pillage it in whole or in part. I don't know how we should decide to use POSIX or SysV semaphores. It looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD based OSes), and POSIX is preferred on Linux. Hmmm, preferred is a tricky word here. I would phrase it slightly differently: POSIX IPC is preferred everywhere (by me, at least) because it's a more modern API. However, SysV IPC is fully supported everywhere while the same can't be said about POSIX IPC. Speaking of POSIX IPC, I also have a posix_ipc module that's quite similar to sysv_ipc and the platform notes in the documentation tell one what support to expect for POSIX IPC under various platforms: http://semanchuk.com/philip/posix_ipc/#platforms Cheers Philip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Under the condition that it is within reason to fix it before the release. What *should* be possible is to disable building SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, multiprocessing locks would stop working on FreeBSD, and concurrent futures; the tests would recognize this lack of features and get skipped. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Hi all, What Victor says above is correct, although I wasn't aware that POSIX IPC under FreeBSD 7.2 was still having problems. Prior to 7.2 it was broken but 7.2 worked OK in my limited testing. In any case, the sysv_ipc module is mine and it's mature and you're welcome to pillage it in whole or in part. The question really is whether you would be willing to port Modules/_multiprocessing/semaphore.c to SysV IPC. I realize that this would be practically new code, but it would be much appreciated. One question is whether it's actually feasible to implement that API with SysV semaphores. I don't know how we should decide to use POSIX or SysV semaphores. It looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD based OSes), and POSIX is preferred on Linux. Hmmm, preferred is a tricky word here. I would phrase it slightly differently: POSIX IPC is preferred everywhere (by me, at least) because it's a more modern API. However, SysV IPC is fully supported everywhere while the same can't be said about POSIX IPC. If you can make the above change, the question then is what API multiprocessing semaphores should be built upon. It seems that you are saying that they should use SysV IPC, and only fall back to POSIX IPC if SysV IPC doesn't work/exist (are there any platforms where this would be the case?) Alternatively, we could have a whitelist of systems on which POSIX IPC is used (==['linux']), and use sysv ipc everywhere else. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, Dec 29, 2010 at 10:28 AM, Martin v. Löwis mar...@v.loewis.de wrote: I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Under the condition that it is within reason to fix it before the release. What *should* be possible is to disable building SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, multiprocessing locks would stop working on FreeBSD, and concurrent futures; the tests would recognize this lack of features and get skipped. Regards, Martin The multiprocessing test suite already skips the tests which use the (broken) functionality on BSD correctly. This logic needs to be added to the concurrent.futures library. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, Dec 29, 2010 at 8:17 AM, Victor Stinner victor.stin...@haypocalc.com wrote: Hi, FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of a concurrent.futures failure. The problem is that test_concurrent_futures uses many (multiprocessing) POSIX semaphores, whereas POSIX semaphores support in FreeBSD is recent and limited. We have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: * http://bugs.python.org/issue10348 * Too many open files errors on x86 FreeBSD 7.2 3.x buildbot ^-- thread in python-dev opened last month I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Does anyone know SysV API? I tried to write a patch, but I never used semaphores (POSIX or SysV). There is a third party module which looks complete and stable: http://semanchuk.com/philip/sysv_ipc/ It is released under the BSD license. It supports semaphores, but also shared memory and message queues. We don't need all of those, semaphores would be enough. I added its author (Philip Semanchuk) to this thread. I don't know how we should decide to use POSIX or SysV semaphores. It looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD based OSes), and POSIX is preferred on Linux. Victor The concurrent.futures tests should (like the multiprocessing test suite) detect the lack of support and skip the tests on the broken platforms. I'm sort of surprised FreeBSD support is still broken in this way though (echoed by Philip) although it could be an issue on that particular buildbot. Moving from POSIX IPC to SYSV should *not* be on the plate for a release blocker - that's a much larger task. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 29.12.2010 18:54, schrieb Jesse Noller: On Wed, Dec 29, 2010 at 10:28 AM, Martin v. Löwis mar...@v.loewis.de wrote: I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Under the condition that it is within reason to fix it before the release. What *should* be possible is to disable building SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, multiprocessing locks would stop working on FreeBSD, and concurrent futures; the tests would recognize this lack of features and get skipped. Regards, Martin The multiprocessing test suite already skips the tests which use the (broken) functionality on BSD correctly. This logic needs to be added to the concurrent.futures library. I'm not so sure that skipping the test is the right approach. Doesn't that mean that the code will still fail at runtime with difficult-to-explain messages? I'd rather prefer if the functionality wasn't available in the first place. Also, what specific test are you referring to? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, 29 Dec 2010 12:58:55 -0500, Jesse Noller jnol...@gmail.com wrote: The concurrent.futures tests should (like the multiprocessing test suite) detect the lack of support and skip the tests on the broken platforms. I'm sort of surprised FreeBSD support is still broken in this way though (echoed by Philip) although it could be an issue on that particular buildbot. If I'm reading the issue correctly, it isn't that it doesn't work, it's that the concurrent.futures tests fail because they create more semaphores than the default FreeBSD install supports. In other words, a user of concurrent.futures really needs to tweak their FreeBSD install to make in fully functional. There should be a way (through sysctl, presumably) to query the maximum number of semaphores and skip the relevant tests on that basis. -- R. David Murray www.bitdance.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 2:14 PM, R. David Murray wrote: On Wed, 29 Dec 2010 12:58:55 -0500, Jesse Noller jnol...@gmail.com wrote: The concurrent.futures tests should (like the multiprocessing test suite) detect the lack of support and skip the tests on the broken platforms. I'm sort of surprised FreeBSD support is still broken in this way though (echoed by Philip) although it could be an issue on that particular buildbot. If I'm reading the issue correctly, it isn't that it doesn't work, it's that the concurrent.futures tests fail because they create more semaphores than the default FreeBSD install supports. In other words, a user of concurrent.futures really needs to tweak their FreeBSD install to make in fully functional. I think that's correct. Furthermore, the default 7.2 install doesn't have the necessary kernel modules loaded in order to use POSIX semaphores. See the notes here for FreeBSD: http://semanchuk.com/philip/posix_ipc/#platforms kldstat on my barely-customized 7.2 installation gives a list of three modules: kernel, acpi.ko and linux.ko (I assume because I asked for Linux executable format compatibility when I installed). bye Philip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 10:43 AM, Martin v. Löwis wrote: Hi all, What Victor says above is correct, although I wasn't aware that POSIX IPC under FreeBSD 7.2 was still having problems. Prior to 7.2 it was broken but 7.2 worked OK in my limited testing. In any case, the sysv_ipc module is mine and it's mature and you're welcome to pillage it in whole or in part. The question really is whether you would be willing to port Modules/_multiprocessing/semaphore.c to SysV IPC. I realize that this would be practically new code, but it would be much appreciated. One question is whether it's actually feasible to implement that API with SysV semaphores. I don't know how we should decide to use POSIX or SysV semaphores. It looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD based OSes), and POSIX is preferred on Linux. Hmmm, preferred is a tricky word here. I would phrase it slightly differently: POSIX IPC is preferred everywhere (by me, at least) because it's a more modern API. However, SysV IPC is fully supported everywhere while the same can't be said about POSIX IPC. If you can make the above change, the question then is what API multiprocessing semaphores should be built upon. It seems that you are saying that they should use SysV IPC, and only fall back to POSIX IPC if SysV IPC doesn't work/exist (are there any platforms where this would be the case?) Actually, I'd probably recommend the opposite. IMO POSIX IPC is better designed and easier to work with so it would be my first choice. SysV would be a fallback for when POSIX IPC is unavailable, broken, incomplete, etc. bye Philip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, 29 Dec 2010 14:14:03 -0500 R. David Murray rdmur...@bitdance.com wrote: On Wed, 29 Dec 2010 12:58:55 -0500, Jesse Noller jnol...@gmail.com wrote: The concurrent.futures tests should (like the multiprocessing test suite) detect the lack of support and skip the tests on the broken platforms. I'm sort of surprised FreeBSD support is still broken in this way though (echoed by Philip) although it could be an issue on that particular buildbot. If I'm reading the issue correctly, it isn't that it doesn't work, it's that the concurrent.futures tests fail because they create more semaphores than the default FreeBSD install supports. Doesn't it suggest a possible resource leak somewhere? Or do the concurrent tests really rely on creating many semaphores? (as opposed to say 5 or 10 of them) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, Dec 29, 2010 at 1:34 PM, Martin v. Löwis mar...@v.loewis.de wrote: Am 29.12.2010 18:54, schrieb Jesse Noller: On Wed, Dec 29, 2010 at 10:28 AM, Martin v. Löwis mar...@v.loewis.de wrote: I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Under the condition that it is within reason to fix it before the release. What *should* be possible is to disable building SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, multiprocessing locks would stop working on FreeBSD, and concurrent futures; the tests would recognize this lack of features and get skipped. Regards, Martin The multiprocessing test suite already skips the tests which use the (broken) functionality on BSD correctly. This logic needs to be added to the concurrent.futures library. I'm not so sure that skipping the test is the right approach. Doesn't that mean that the code will still fail at runtime with difficult-to-explain messages? I'd rather prefer if the functionality wasn't available in the first place. Also, what specific test are you referring to? Regards, Martin If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Le mercredi 29 décembre 2010 à 21:04 +0100, Antoine Pitrou a écrit : Doesn't it suggest a possible resource leak somewhere? I already checked that: all locks are destroyed correctly on each test. - test_all_completed_some_already_completed() uses 51 SemLock objects - test_first_completed() uses 41 SemLock objects - test_processes_terminate() uses 41 SemLock objects - etc. FreeBSD has an arbitrary limit of 30 semaphores. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
If you can make the above change, the question then is what API multiprocessing semaphores should be built upon. It seems that you are saying that they should use SysV IPC, and only fall back to POSIX IPC if SysV IPC doesn't work/exist (are there any platforms where this would be the case?) Actually, I'd probably recommend the opposite. IMO POSIX IPC is better designed and easier to work with so it would be my first choice. better designed and easier to work with don't count if you have to do both, anyway. The code will be ugly - so what, once it is written? It's not being exposed to the user. SysV would be a fallback for when POSIX IPC is unavailable, broken, incomplete, etc. The problem seems to be that it is not possible to reliably determine these cases. In particular, the ridiculous low limits seem to be difficult to detect, and it's not clear what a required minimum level should be. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
The multiprocessing test suite already skips the tests which use the (broken) functionality on BSD correctly. This logic needs to be added to the concurrent.futures library. Also, what specific test are you referring to? Can you kindly point me to the test that skips if broken functionality on BSD is detected? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 3:49 PM, Martin v. Löwis mar...@v.loewis.de wrote: If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better). Whatever you choose to say Martin. It does work, and is supported to the limitations that FreeBSD imposes. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 29.12.2010 22:34, schrieb Jesse Noller: On Dec 29, 2010, at 3:49 PM, Martin v. Löwis mar...@v.loewis.de wrote: If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better). Whatever you choose to say Martin. It does work, and is supported to the limitations that FreeBSD imposes. So what do you propose to do? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 4:54 PM, Martin v. Löwis mar...@v.loewis.de wrote: Am 29.12.2010 22:34, schrieb Jesse Noller: On Dec 29, 2010, at 3:49 PM, Martin v. Löwis mar...@v.loewis.de wrote: If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better). Whatever you choose to say Martin. It does work, and is supported to the limitations that FreeBSD imposes. So what do you propose to do? I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me. I would document the limitation in the futures/multiprocessing documentation, and skip that particular test for now on FreeBSD. FreeBSDs limitations seem arbitrary, but I'm not a FBSD expert. This isn't a bug in either futures or multiprocessing though. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me. And indeed, I don't propose to make that a release blocker. Instead, I propose to disable support for the module (either multiprocessing or concurrent.futures only) on FreeBSD, and make such disabling a release blocker. I would document the limitation in the futures/multiprocessing documentation, and skip that particular test for now on FreeBSD. Just skipping the test case seems inappropriate, when it really tests the core functionality of the module in question. The failing test really means that the module does not work on the system. FreeBSDs limitations seem arbitrary, but I'm not a FBSD expert. This isn't a bug in either futures or multiprocessing though. I don't claim that it's a bug. I claim that FreeBSD is just not supported (saying that FreeBSD doesn't support the multiprocessing module might be more appropriate). Somebody will have to correctly fix it - either the FreeBSD people, or the multiprocessing authors, or the concurrent.future authors. Meanwhile, we must admit that it just doesn't work (or else the test case would not fail). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 5:24 PM, Martin v. Löwis wrote: I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me. And indeed, I don't propose to make that a release blocker. Instead, I propose to disable support for the module (either multiprocessing or concurrent.futures only) on FreeBSD, and make such disabling a release blocker. Note that we only know that the test fails on FreeBSD 7.2. The current release is FreeBSD 8.1. The 8.x series might work just fine. I don't know which test is failing under 7.2. Has anyone tested it under 7.3 or 8.x? bye Philip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, 29 Dec 2010 23:24:32 +0100 Martin v. Löwis mar...@v.loewis.de wrote: I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me. And indeed, I don't propose to make that a release blocker. Instead, I propose to disable support for the module (either multiprocessing or concurrent.futures only) on FreeBSD, and make such disabling a release blocker. I don't really agree with this. There's no need to explicitly forbid use of multiprocessing from FreeBSD. First, it is not our task to validate that each and every OS conforms to the APIs it claims to implement. Second, such disabling would make life uselessly more complicated for users the day FreeBSD actually fixes their stuff. So, IMO, skipping tests is enough. At worse, multiprocessing can issue a warning when imported under these OSes. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Martin v. Löwis mar...@v.loewis.de writes: I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me. And indeed, I don't propose to make that a release blocker. Instead, I propose to disable support for the module (either multiprocessing or concurrent.futures only) on FreeBSD, and make such disabling a release blocker. Or, I'll make the same offer I think I made in the multiprocessing case, which is I can build a kernel on the buildbot with a higher limit, if that's needed just to ensure test completion. Yes, it would also mean that users would need to do the same (or in later FreeBSD releases it can be done dynamically via sysctl), but that didn't seem to be considered a big hurdle in the prior discussion for multiprocessing. That would essentially switch this to a documentation issue, to document that on older FreeBSD platforms a user needs to take some steps, either as a startup configuration, or a kernel rebuild depending on release. That assumes that normal use of the module will need as many semaphores as the tests, but if not, presumably the documentation comments could become more a caution than a requirement. I've also been considering retiring the 6.x buildbot in favor of an 8.x (I only have resources for 2, and even that is slow). Updating the 7.x buildbot would also include dynamic adjustment of the limit, which I think based on the prior discussion her was mentioned as being in 7.3. Though 7.4 releases late in January so could jump right to that. -- David PS: In regards to another comment in this thread, while 7.2 doesn't load the POSIX support by default, the buildbot does have the appropriate loader configuration to include it. PPS: I am about to be traveling on business through 1/10, so unlikely to make any buildbot changes before then. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Thu, Dec 30, 2010 at 8:33 AM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 29 Dec 2010 23:24:32 +0100 Martin v. Löwis mar...@v.loewis.de wrote: I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me. And indeed, I don't propose to make that a release blocker. Instead, I propose to disable support for the module (either multiprocessing or concurrent.futures only) on FreeBSD, and make such disabling a release blocker. I don't really agree with this. There's no need to explicitly forbid use of multiprocessing from FreeBSD. First, it is not our task to validate that each and every OS conforms to the APIs it claims to implement. Second, such disabling would make life uselessly more complicated for users the day FreeBSD actually fixes their stuff. Also, I believe you can get it to work on FreeBSD 7.2 by fiddling with the arbitrary limits. So +1 for skipping the affected tests on FreeBSD so the buildbot state can tell us something useful (conditionally wrapping them with expected failure may be even better, as then we'll find out when the FreeBSD out-of-the-box configuration actually supports running them). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Le mercredi 29 décembre 2010 à 21:49 +0100, Martin v. Löwis a écrit : Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. More numbers (on Linux): - Queue: 3 SemLock - Condition: 4 SemLock - Event: 5 SemLock - Call (test_concurrent_futures): 10 SemLock (2 Event) - ProcessPoolExecutor: 11 SemLock (2 Queue, 1 Condition) FreeBSD 7.2 is limited to 30 semaphores, so with ProcessPoolExecutor, you can only create *one* Call object, whereas some tests use 4 Call objects or more. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On 12/29/2010 5:44 PM, David Bolen wrote: Or, I'll make the same offer I think I made in the multiprocessing case, which is I can build a kernel on the buildbot with a higher limit, if that's needed just to ensure test completion. Yes, it would also mean that users would need to do the same (or in later FreeBSD releases it can be done dynamically via sysctl), but that didn't seem to be considered a big hurdle in the prior discussion for multiprocessing. That would essentially switch this to a documentation issue, to document that on older FreeBSD platforms a user needs to take some steps, either as a startup configuration, or a kernel rebuild depending on release. Testing on updated buildbots and documenting older BSD limits and fixes seems like the best solution. Martin is right that we really do want the tests to run to catch any real bugs (on our side). -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 5:46 PM, Nick Coghlan wrote: On Thu, Dec 30, 2010 at 8:33 AM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 29 Dec 2010 23:24:32 +0100 Martin v. Löwis mar...@v.loewis.de wrote: I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me. And indeed, I don't propose to make that a release blocker. Instead, I propose to disable support for the module (either multiprocessing or concurrent.futures only) on FreeBSD, and make such disabling a release blocker. I don't really agree with this. There's no need to explicitly forbid use of multiprocessing from FreeBSD. First, it is not our task to validate that each and every OS conforms to the APIs it claims to implement. Second, such disabling would make life uselessly more complicated for users the day FreeBSD actually fixes their stuff. Also, I believe you can get it to work on FreeBSD 7.2 by fiddling with the arbitrary limits. So +1 for skipping the affected tests on FreeBSD so the buildbot state can tell us something useful (conditionally wrapping them with expected failure may be even better, as then we'll find out when the FreeBSD out-of-the-box configuration actually supports running them). A bit of info about FreeBSD 7.2 and 8.0 (the only versions I have installed). My installs are set up explicitly for testing and so they have had very little customization. - `sysctl -a|grep p1003` displays the config values specific to POSIX IPC. I think `sysctl -a|grep ipc` shows values for SysV IPC. It looks like the POSIX 1003_1b names in sysctl are similar to the names here (e.g. SEM_NSEMS_MAX, SEM_VALUE_MAX, etc.) -- http://pubs.opengroup.org/onlinepubs/009695399/basedefs/limits.h.html - Under my 7.2, sysctl.p1003_1b.sem_nsems_max == 0 and I can't create semaphores with my posix_ipc extension. After `kldload sem`, sysctl.p1003_1b.sem_nsems_max == 30 and my posix_ipc semaphore shared mem demo works. - Under my 8.0, sysctl.p1003_1b.sem_nsems_max == 30 and my posix_ipc semaphore shared mem demo works even without `kldload sem`. `kldstat` doesn't show sem.ko as loaded, so I assume that POSIX semaphores are now a default part of the kernel. - Under my 7.2 *and* 8.0, sysctl.p1003_1b.mq_open_max == 0 and I can't create a message queue with my posix_ipc extension. After `kldload mqueuefs`, sysctl.p1003_1b.mq_open_max remains 0 (huh?) but my posix_ipc message queue demo works. - Under my 8.0, sysctl.p1003_1b.sem_nsems_max is read only even if I change it in /etc/sysctl.conf. According to this post, sem_nsems_max doesn't become dynamic until 8.1: http://www.gossamer-threads.com/lists/python/dev/875319?do=post_view_threaded Hope this helps a bit Philip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 2:55 PM, Victor Stinner wrote: Le mercredi 29 décembre 2010 à 21:49 +0100, Martin v. Löwis a écrit : Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. More numbers (on Linux): - Queue: 3 SemLock - Condition: 4 SemLock - Event: 5 SemLock - Call (test_concurrent_futures): 10 SemLock (2 Event) - ProcessPoolExecutor: 11 SemLock (2 Queue, 1 Condition) FreeBSD 7.2 is limited to 30 semaphores, so with ProcessPoolExecutor, you can only create *one* Call object, whereas some tests use 4 Call objects or more. Great detective work! This would suggest that ProcessPoolExecutors are useable on FreeBSD 7.2 so long as the user doesn't create more than two at once (which probably isn't a big deal for most apps). So skipping the test is probably the way to go. Cheers, Brian Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 12:49 PM, Martin v. Löwis wrote: If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. I actually think that my tests may be overdone - in order to probe for specific race conditions they use a lot of locks to force calls to complete in a specific order. I'm thinking about pairing the tests down to only demonstrate basic correctness. This should fix the tests on FreeBSD and Windows. Then, when Python 3.2 is released, I can gradually introduce more comprehensive tests while ensuring that I keep the buildbots green on all supported platforms. Thoughts? Cheers, Brian So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
