[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
STINNER Victor victor.stin...@haypocalc.com added the comment: (Victor, please don't file many bugs in a single issue!) I thought that these issues were the same. Victor -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Charles-Francois Natali neolo...@free.fr added the comment: I think those lockups are due to a race in the Pool shutdown code. In Lib/multiprocessing/pool.py: def close(self): debug('closing pool') if self._state == RUN: self._state = CLOSE self._worker_handler._state = CLOSE self._taskqueue.put(None) We set the current state to CLOSE, and send None to the taskqueue, so that task_handler detects that we want to shut down the queue and sends None (sentinel) to the inqueue for each worker process. When a worker process receives this sentinel, it exists, and when Pool's join method is called, each process is joined successfully. Now, there's a problem, because of the worker_hanler thread. This thread constantly starts new threads if existing one exited after having completed their work: def _handle_workers(pool): while pool._worker_handler._state == RUN and pool._state == RUN: pool._maintain_pool() time.sleep(0.1) debug('worker handler exiting') where def _maintain_pool(self): Clean up any exited workers and start replacements for them. if self._join_exited_workers(): self._repopulate_pool() Imagine the following happens: worker_handler checks that the pool is still running (state == RUN), but before calling maintain_pool, it's preempted (releasal of the GIL), and Pool's close() methode is called : state is set to CLOSE, None is put to taskqueue, and worker threads exit. Then, Pool's join is called: def join(self): debug('joining pool') assert self._state in (CLOSE, TERMINATE) self._worker_handler.join() self._task_handler.join() self._result_handler.join() for p in self._pool: p.join() this blocks until worker_handler exits. This thread sooner or later resumes and calls maintain_pool. maintain_pool calls repopulate_pool, which recreates new worker threads/processes. Then, worker_handler checks the current state, sees CLOSE, and exists. Then, Pool's join blocks there: for p in self._pool: p.join() since the newly created processes never receive the sentinels (already consumed by the previous worker processes)... This race can be reproduced almost every time by just adding: def _handle_workers(pool): while pool._worker_handler._state == RUN and pool._state == RUN: +time.sleep(1) pool._maintain_pool() time.sleep(0.1) debug('worker handler exiting') Then something as simple as this will block: p = multiprocessing.Pool(3) p.close() p.join() I still have to think of a clean way to solve this. -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Charles-Francois Natali neolo...@free.fr added the comment: Attached is a patch fixing this race, and a similar one in Pool's terminate. -- keywords: +patch Added file: http://bugs.python.org/file21608/pool_shutdown_race.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___diff -r bbfc65d05588 Lib/multiprocessing/pool.py --- a/Lib/multiprocessing/pool.py Thu Apr 07 10:48:29 2011 -0400 +++ b/Lib/multiprocessing/pool.py Sun Apr 10 23:52:22 2011 +0200 @@ -322,6 +322,8 @@ while pool._worker_handler._state == RUN and pool._state == RUN: pool._maintain_pool() time.sleep(0.1) +# send sentinel to stop workers +pool._taskqueue.put(None) debug('worker handler exiting') @staticmethod @@ -440,7 +442,6 @@ if self._state == RUN: self._state = CLOSE self._worker_handler._state = CLOSE -self._taskqueue.put(None) def terminate(self): debug('terminating pool') @@ -474,7 +475,6 @@ worker_handler._state = TERMINATE task_handler._state = TERMINATE -taskqueue.put(None) # sentinel debug('helping task handler/workers to finish') cls._help_stuff_finish(inqueue, task_handler, len(pool)) @@ -484,6 +484,11 @@ result_handler._state = TERMINATE outqueue.put(None) # sentinel +# we must wait for the worker handler to exit before terminating +# workers because we don't want workers to be restarted behind our back +debug('joining worker handler') +worker_handler.join() + # Terminate workers which haven't already finished. if pool and hasattr(pool[0], 'terminate'): debug('terminating workers') ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Antoine Pitrou pit...@free.fr added the comment: Nice! See also issue11814. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Roundup Robot devnull@devnull added the comment: New changeset d5e43afeede6 by Antoine Pitrou in branch '3.2': Issue #8428: Fix a race condition in multiprocessing.Pool when terminating http://hg.python.org/cpython/rev/d5e43afeede6 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Roundup Robot devnull@devnull added the comment: New changeset dfc61dc14f59 by Antoine Pitrou in branch '2.7': Issue #8428: Fix a race condition in multiprocessing.Pool when terminating http://hg.python.org/cpython/rev/dfc61dc14f59 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Antoine Pitrou pit...@free.fr added the comment: Should be fixed now, thank you Charles-François. As for the TestCondition failure, there's a separate issue11790 open. (Victor, please don't file many bugs in a single issue!) -- resolution: - fixed stage: - committed/rejected status: open - closed versions: +Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
STINNER Victor victor.stin...@haypocalc.com added the comment: Another test_multiprocessing failure (30 min timeout) on x86 XP-4 3.x: -- [125/354] test_multiprocessing Thread 0x0e5c: File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 235 in wait File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\queue.py, line 185 in get File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\pool.py, line 372 in _handle_results File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 688 in run File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 735 in _bootstrap_inner File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 708 in _bootstrap Thread 0x0c14: File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 235 in wait File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\queue.py, line 185 in get File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\pool.py, line 331 in _handle_tasks File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 688 in run File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 735 in _bootstrap_inner File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 708 in _bootstrap Thread 0x0d78: File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\pool.py, line 324 in _handle_workers File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 688 in run File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 735 in _bootstrap_inner File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 708 in _bootstrap Thread 0x03b8: File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 235 in wait File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\queue.py, line 185 in get File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\pool.py, line 102 in worker File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 688 in run File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 735 in _bootstrap_inner File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 708 in _bootstrap Thread 0x01e8: File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 235 in wait File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\queue.py, line 185 in get File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\pool.py, line 102 in worker File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 688 in run File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 735 in _bootstrap_inner File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 708 in _bootstrap Thread 0x02ac: File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 235 in wait File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\queue.py, line 185 in get File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\pool.py, line 102 in worker File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 688 in run File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 735 in _bootstrap_inner File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 708 in _bootstrap Thread 0x05a4: File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 235 in wait File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\queue.py, line 185 in get File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\pool.py, line 102 in worker File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 688 in run File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 735 in _bootstrap_inner File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\threading.py, line 708 in _bootstrap Thread 0x0c84: File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\forking.py, line 287 in wait File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\process.py, line 149 in join File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\pool.py, line 458 in join File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_multiprocessing.py, line 1143 in test_async_error_callback File
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
STINNER Victor victor.stin...@haypocalc.com added the comment: Here is a nice trace on PPC Leopard 3.x thanks to faulthandler + regrtest timeout (30 minutes): --- ... [218/354] test_multiprocessing Thread 0xf0617000: File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 235 in wait File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/queue.py, line 185 in get File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/multiprocessing/pool.py, line 372 in _handle_results File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 688 in run File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 735 in _bootstrap_inner File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 708 in _bootstrap Thread 0xf0595000: File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 235 in wait File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/queue.py, line 185 in get File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/multiprocessing/pool.py, line 331 in _handle_tasks File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 688 in run File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 735 in _bootstrap_inner File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 708 in _bootstrap Thread 0xf0513000: File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/multiprocessing/pool.py, line 324 in _handle_workers File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 688 in run File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 735 in _bootstrap_inner File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 708 in _bootstrap Thread 0xf0491000: File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 235 in wait File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/queue.py, line 185 in get File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/multiprocessing/pool.py, line 102 in worker File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 688 in run File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 735 in _bootstrap_inner File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 708 in _bootstrap Thread 0xf040f000: File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 235 in wait File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/queue.py, line 185 in get File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/multiprocessing/pool.py, line 102 in worker File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 688 in run File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 735 in _bootstrap_inner File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 708 in _bootstrap Thread 0xf038d000: File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 235 in wait File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/queue.py, line 185 in get File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/multiprocessing/pool.py, line 102 in worker File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 688 in run File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 735 in _bootstrap_inner File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 708 in _bootstrap Thread 0xf030b000: File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 235 in wait File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/queue.py, line 185 in get File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/multiprocessing/pool.py, line 102 in worker File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 688 in run File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 735 in _bootstrap_inner File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/threading.py, line 708 in _bootstrap Thread 0xa09e1820: File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/multiprocessing/forking.py, line 134 in poll File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/multiprocessing/forking.py, line 149 in wait File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/multiprocessing/process.py, line 149 in join File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/multiprocessing/pool.py, line 458 in join File /Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/test/test_multiprocessing.py, line 1195 in test_pool_worker_lifetime File
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Florent Xicluna florent.xicl...@gmail.com added the comment: Seen on Windows 7 3.1: test test_multiprocessing failed -- Traceback (most recent call last): File D:\cygwin\home\db3l\buildarea\3.1.bolen-windows7\build\lib\test\test_multiprocessing.py, line 746, in test_notify_all self.assertReturnsIfImplemented(6, get_value, woken) File D:\cygwin\home\db3l\buildarea\3.1.bolen-windows7\build\lib\test\test_multiprocessing.py, line 120, in assertReturnsIfImplemented return self.assertEqual(value, res) AssertionError: 6 != 2 There's a second issue at the end of the log, it seems to be the case 3 of issue #9592 (RuntimeError: maximum recursion depth exceeded while calling a Python object). http://www.python.org/dev/buildbot/all/builders/x86%20Windows7%203.1/builds/676 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Jesse Noller jnol...@gmail.com added the comment: Is this intermittent, or consistently failing? Updating it with more buildbot failures doesn't help. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Florent Xicluna florent.xicl...@gmail.com added the comment: It is intermittent on most buildbots. The exception is x86 FreeBSD 7.2 3.x where it occurs on each run. http://www.python.org/dev/buildbot/all/builders/x86%20FreeBSD%207.2%203.x/ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Florent Xicluna florent.xicl...@gmail.com added the comment: Still failing on 3.2 and 2.7 - x86 FreeBSD 7.2 3.x r83981, r83971 ... http://www.python.org/dev/buildbot/all/builders/x86%20FreeBSD%207.2%203.x/builds/823 - x86 FreeBSD 7.2 2.7 r83985, r83806 ... http://www.python.org/dev/buildbot/all/builders/x86%20FreeBSD%207.2%202.7/builds/67 - x86 XP-4 2.7 r83985 http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%202.7/builds/146 - sparc Ubuntu 2.7 r82939 http://www.python.org/dev/buildbot/all/builders/sparc%20Ubuntu%202.7/builds/11 Traceback on x86 XP-4 2.7: test test_multiprocessing failed -- Traceback (most recent call last): File D:\cygwin\home\db3l\buildarea\2.7.bolen-windows\build\lib\test\test_multiprocessing.py, line 746, in test_notify_all self.assertReturnsIfImplemented(6, get_value, woken) File D:\cygwin\home\db3l\buildarea\2.7.bolen-windows\build\lib\test\test_multiprocessing.py, line 120, in assertReturnsIfImplemented return self.assertEqual(value, res) AssertionError: 6 != 5 Traceback on x86 FreeBSD 7.2 3.x: == FAIL: test_notify_all (test.test_multiprocessing.WithProcessesTestCondition) -- Traceback (most recent call last): File /usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py, line 740, in test_notify_all self.assertReturnsIfImplemented(6, get_value, woken) File /usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py, line 114, in assertReturnsIfImplemented return self.assertEqual(value, res) AssertionError: 6 != 3 Additionally, these 2.7 buildbots hang on test_multiprocessing: - PPC Leopard 2.7 r83996, r83961 ... http://www.python.org/dev/buildbot/all/builders/PPC%20Leopard%202.7/builds/84 - PPC Tiger 2.7 r83961, r83879 ... http://www.python.org/dev/buildbot/all/builders/PPC%20Tiger%202.7/builds/100 - x86 FreeBSD 7.2 2.7 r83765 http://www.python.org/dev/buildbot/all/builders/x86%20FreeBSD%207.2%202.7/builds/54 -- components: +Tests nosy: +flox type: - behavior versions: +Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Changes by STINNER Victor victor.stin...@haypocalc.com: -- title: buildbot: test_multiprocessing timeout - buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?) ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Changes by STINNER Victor victor.stin...@haypocalc.com: -- nosy: +jnoller ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8428 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com