[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Davin Potts added the comment: In some (but not necessarily all) circumstances, the multiprocessing module is now (in 2.7 and 3.x) able to detect the infinite spawning behavior described due to unsafe importing of the main module on Windows. The resulting error message looks like this: RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The freeze_support() line can be omitted if the program is not going to be frozen to produce an executable. Hopefully this behavior and the resulting message will both alert developers when such a problem is introduced in a project and communicate to them what the problem is. Giant blinking red letters aside, hopefully this pragmatic solution will do a better job of educating and helping developers notice and deal with such issues. In the combinatoric explosion of possibilities that combine nosetest-binaries with PyDev with other useful dev tools, if this error message is being suppressed or otherwise not shared with the end user of those tools, that would warrant opening an issue with that tool. -- nosy: +davin resolution: - fixed stage: - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Changes by Mark Lawrence breamore...@yahoo.co.uk: -- nosy: +sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Changes by Martin Dengler mar...@martindengler.com: -- nosy: +mdengler ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Matt Chaput added the comment: IIRC the root issue turned out to be that when you execute any multiprocessing statements at the module/script level on Windows, you need to put it under if __name__ == __main__, otherwise it will cause infinite spawning. I think this is mentioned in the multiprocessing docs but should probably be in giant blinking red letters ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Martin Dengler added the comment: the root issue turned out to be that when you execute any multiprocessing statements at the module/script level on Windows, you need to put it under if __name__ == __main__, otherwise it will cause infinite spawning. Same for me. The error message and failure mode are completely unhelpful, though. I think this is mentioned in the multiprocessing docs but should probably be in giant blinking red letters ;) Indeed. It would be even better if I or someone else had time to contribute a patch to fix the behaviour and, or at least the failure mode / error message. In a large codebase with multiple contributors it might not be so simple to track down the commit that caused the issue, especially if one is just starting out and the tests aren't clean. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Chris Jones ch...@chrisejones.com added the comment: You can work around this issue by using: python.exe -c import nose; nose.main() instead of nosetests Note that nose.main() with no args parses sys.argv -- nosy: +Chris.Jones ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Changes by David Barnett davidbarne...@gmail.com: -- nosy: +mu_mind ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Matt Chaput m...@whoosh.ca added the comment: I don't know what to tell you... to the best of my knowledge there's absolutely no way for my code to kick off the entire test suite -- I always do that through PyDev (which doesn't cause the bug, by the way). The closest thing is the boilerplate at the bottom of every test file: if __name__ == __main__: unittest.main() ...but even that would only start the tests in that file, not the entire suite. Another thing that makes me think multiprocessing is re-running the original command line is that if I use python setup.py test to start the tests, when it gets to the MP tests it seems to run that command for each Process that gets started, but if I use nosetests, it seems to run nosetests for each started Process. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: Nose works correctly for me with multiprocessing. In a directory, I have: == run_nose.py = from nose import main if __name__ == '__main__': main() == test_me.py == from multiprocessing import Pool import os, time def foo(x): time.sleep(0.1) return (x, os.getpid()) def test_me(): pool = Pool(processes=4) x = pool.map(foo, range(10)) a, b = zip(*x) print a, b assert list(a) == range(10) assert 1 len(set(b)) = 4 Now when I do: c:\python27\python run_nose.py the test runs correctly. Can you try this test in your environment? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Matt Chaput m...@whoosh.ca added the comment: If I do c:\python27\python run_nose.py it works correctly. If I do nosetests I get the process explosion. Maybe the bug is in how distutils and nose work from the command line? I'm confused. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: Ah, is 'nosetests' a .exe file? A frozen executable? In this case, can you try the solution proposed at http://docs.python.org/library/multiprocessing.html#multiprocessing.freeze_support -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Antoine Pitrou pit...@free.fr added the comment: If I do c:\python27\python run_nose.py it works correctly. If I do nosetests I get the process explosion. Maybe the bug is in how distutils and nose work from the command line? I'm confused. The bug is in nose itself. You should report a bug there. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
New submission from Matt Chaput m...@whoosh.ca: If you start unit tests with a command line such as python setup.py test or nosetests, if the tested code starts a multiprocessing.Process on Windows, each new process will act as if it was started as python setup.py test/nosetests, leading to an infinite explosion of processes that eventually locks up the entire machine. -- components: Windows messages: 128768 nosy: mattchaput priority: normal severity: normal status: open title: Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows type: behavior versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: Using multiprocessing on Windows can be different; please read http://docs.python.org/library/multiprocessing.html#windows especially the part named Safe importing of main module. On Windows, fork() does not exist, so a new interpreter must be started, which will import the current module; this must not start the test suite again! Adding if __name__ == '__main__' somewhere is probably the solution. If not, you should move the target function to another module. -- nosy: +amaury.forgeotdarc ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Matt Chaput m...@whoosh.ca added the comment: Thank you, I understand all that, but I don't think you understand the issue. My code is not __main__. I am not starting the test suite. It's the distutils/nose code that's doing that. It seems as if the multiprocessing module is starting new Windows processes by duplicating the command line of the original process. That doesn't seem to work very well, given the example of running test suites, hence the bug. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: It seems as if the multiprocessing module is starting new Windows processes by duplicating the command line of the original process. It does not. The spawned processes use the command:: python.exe -c 'from multiprocessing.forking import main; main()' --multiprocessing-fork [handle#] And only after, the multiprocessing machinery overrides sys.argv with the same value as the initial process. There is certainly some code in one of your modules that starts running the tests. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
