[issue17555] Creating new processes after importing multiprocessing.managers consumes more and more memory
Charles-François Natali added the comment: Richard, your patch is indeed better, and fixes the problem. With a test it should be ready to go in. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17555] Creating new processes after importing multiprocessing.managers consumes more and more memory
Charles-François Natali added the comment: That's due to a leak in forker-registered handlers: The _afterfork_registry is never cleared, so spawning processes recursively keeps feeling it, which ends up consuming a huge amount of memory and slowing process creation greatly. Could you try the attached patch? -- keywords: +patch nosy: +neologix Added file: http://bugs.python.org/file29592/afterfork.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17555] Creating new processes after importing multiprocessing.managers consumes more and more memory
Changes by Charles-François Natali cf.nat...@gmail.com: -- nosy: +sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17555] Creating new processes after importing multiprocessing.managers consumes more and more memory
Richard Oudkerk added the comment: It seems to be a problem with ForkAwareThreadLock. Could you try the attached patch? -- Added file: http://bugs.python.org/file29593/forkawarethreadlock.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17555] Creating new processes after importing multiprocessing.managers consumes more and more memory
Richard Oudkerk added the comment: _afterfork_registry is not supposed to be cleared. But the problem with ForkAwareThreadLocal meant that the size of the registry at generation n is 2**n! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17555] Creating new processes after importing multiprocessing.managers consumes more and more memory
New submission from Marc Brünink: Importing multiprocessing.managers and creating new processes recursively is not a good idea. You will run out of memory very fast. On my machine with 8GB it only takes about 20 forks until I start swapping. -- components: Library (Lib) files: test_fork_02.py messages: 185315 nosy: Marc.Brünink priority: normal severity: normal status: open title: Creating new processes after importing multiprocessing.managers consumes more and more memory type: resource usage versions: Python 2.7, Python 3.2 Added file: http://bugs.python.org/file29590/test_fork_02.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
