Aaron <athompson6...@gmail.com> added the comment:
I ran into this bug answering this question on Stack Overflow: https://stackoverflow.com/questions/68890437/cannot-use-result-from-multiprocess-pool-directly I have minimized the code required to replicate the behavior, but it boils down to: when using "spawn" to create a multiprocessing pool, if an exception occurs during the bootstrapping phase of the new child or during the initialization function with any start method, it is just cleaned up, and another takes its place (which will also fail). This creates an infinite loop of creating child workers, workers exiting due to an exception, and re-populating the pool with new workers. ``` import multiprocessing multiprocessing.set_start_method("spawn") # bootstraping only problem with spawn def task(): print("task") if __name__ == "__main__": with multiprocessing.Pool() as p: p.apply(task) else: raise Exception("raise in child during bootstraping phase") ################################################# # or ################################################# import multiprocessing # multiprocessing.set_start_method("fork") # fork or spawn doesn't matter def task(): print("task") def init(): raise Exception("raise in child during initialization function") if __name__ == "__main__": with multiprocessing.Pool(initializer=init) as p: p.apply(task) ``` If Pool._join_exited_workers could determine if a worker exited before bootstrapping, or the initialization function completed, It would indicate a likely significant problem. I'm fine with exceptions in the worker target function not being re-raised in the parent, however it seems the Pool should stop trying if it's failing to create new workers. ---------- nosy: +athompson6735 versions: +Python 3.9 -Python 3.8 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue43306> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
