On Saturday 06 November 2010 22:36:44 you wrote: > I couldn't find the matching failures that you're talking about, but > then I figured out you mean the FreeBSD7 (7.2) buildbot, not the > FreeBSD (6.4) buildbot ....
Search "test_concurrent_futures" in: http://www.python.org/dev/buildbot/builders/x86%20FreeBSD%207.2%203.x/builds/1154/steps/test/logs/stdio I specified "x86 FreeBSD 7.2 3.x" in the email title. > (...) > I noticed that the failures seem to always be on a semaphore call. > Some quick googling found a few references that seems to imply that > the number of posix semaphores are very limited (like 30), and can't > be changed without recompiling the kernel from source. So that's not > so big a threshold for the tests to have perhaps started crossing > since issue7272 was fixed. Certainly seems more likely than 3000+ > files or 1500+ processes. Nice catch. The problem is the total number of semaphores: I reproduced the bug in my FreeBSD 8 VM. The first test fails at the creation of the 31th semaphore. The first failing test if test_all_completed. And it looks like this test doesn't destroy the semaphore at exit: my counter (based on __init__/__del__) is still at 15 when exiting the test! > I wonder if it's possible to deduce if this started recently or not? > The web buildbot interface doesn't go back that far, and an additional > complexity is that the FreeBSD builds tend to have various errors > somewhat consistently over time, but perhaps there are server logs we > can grep for this particular error? No idea. > Not sure if the best approach at this point is to see if the tests can > use fewer semaphores, skip these tests under FreeBSD 7 like 6, or if > it's important enough to compile a new kernel with a higher semaphore > limit. You wrote that the "POSIX" semaphore are very limited. Do it mean that there is another kind of semaphore will an higher limit? I don't think that skipping the test is a good idea: it looks like a real bug in (a limitation of) the ProcessPoolExecutor implementation on FreeBSD. Eg. test_map fails on FreeBSD 7.2 with ProcessPoolExecutorTest which uses self.executor = futures.ProcessPoolExecutor(max_workers=1): only one worker process! It looks like it is possible to tune semaphore limits on FreeBSD, without recompiling the kernel, by using boot loader option (kern.ipc.sem* options). But ask the FreeBSD user to tune its boot loader options to use the concurrent.futures module is not pratical :-) Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com