[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
Sincerely I would have to agree that it's seems a bit excessive the `cancel_on_error`, unless it enabled by default and implemented in the abstract class it should probably not be included, that was just an idea to keep backwards compatibility. I will personally simply add subclass of my prefered executor with the following for my particular use case: def __exit__(self, exc_type, exc_val, exc_tb): self.shutdown(wait=True, cancel_futures=exc_val is not None) return False Again, I think aborting futures on uncaught exceptions make sense, but it would break backwards compatibility. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/3XDUZPZS3VCGX2PJMZHILI2L65V7BEGS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
> But, it would potentially risk adding an underutilized parameter to the > executor constructor (which contributes to feature bloat). That's true, personally I would always enable cancel_on_error (making it redundant and implementing it in the abstract class), but that's just my use case. You can always add that functionality later with your own subclass. Personally I think it make sense to do it by default but that's just me and it also changes current behaviour. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/MMS3OGNA6XNKT63DYWVB4DZFHESLKOXM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
> Hmm, it should be possible. Do you specifically mean cancelling the pending > futures once a single one of the submitted functions raises an exception, > or cancelling the pending futures when the Executor itself raises an > exception (I.E. BrokenProcessPool)? I would assume the prior, since that > seems more useful to me. I was referring to any exception within the context manager of the executor, not the futures or just the executor's exceptions, something such as the following: with ThreadPoolExecutor(cancel_on_error=True) as executor: # Code here raise Exception Then if Executor.__exit__ detects an exception it would call shutdown with cancel_futures set to True. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/OAMG4JUFK54FTAASHQG5WNAA7BU7KRS7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] concurrent.futures deinitializer for Executor
Hi, I thought maybe a callback before the worker of a Executor finishes would be helpful. It would allow the setup and teardown resources (such as file, databases, connections, ...) across workers. My particular use case is for closing down requests.Session objects, that currently its very complicated to cleanup properly. There are many other use cases, a generic format would be something like the following: import threading from concurrent.futures import ThreadPoolExecutor worker_local = threading.local() def session_initializer(): # Initialize costly resources for a particular worker worker_local.resources = ... def session_deinitializer(): # Deinitialize worker resources carefully worker_local.resources ... with ThreadPoolExecutor(initializer=session_initializer, deinitializer=session_deinitializer) as executor: executor.submit(task) ... ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/BOKCVLL4B7COBVXFS5BPTTFTRFU3JKJD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
Thank you so much for the work, I was very confused on how to even start implementing it in the ProcessPoolExecutor, but you finished everything super quick! I'm suppose that this might be better in another thread but... Could it be possible to changing the context manager of the executor to cancel futures on uncaught exceptions? If not I could also be added as an optional parameter to the executor constructor, "cancel_on_error" set to False to not changing anything. Personally I was thinking of using the executor like that; currently I have to either place everything in a try except block, handle it and then reraise it or subclass each executor. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/R5VMWP75K35KF3DJN2F5MBZXD5UOQIV3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
> It looks like you have a good handle on the code -- do you want to submit a > PR to GitHub to add such a parameter? Thanks, but I'm not really sure how to implement it in the ProcessPoolExecutor, I just think the solution is probably related to the code responsible of handling a failed initializer (since they do very similar things). On the ThreadPoolExecutor maybe I could, but I haven't really checked for side effects or weird things. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/FGTSBGTXW63SSXFQBASLCTXAT4M6RGGN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
> But I don’t think “terminate” is the right name. Maybe “cancel”? Or even > “shutdown(wait=whatever, cancel=True)?” "terminate" was definitely not a good name, especially because it doesn't actually terminate anything, it just cancels some of the operations. Since it also has to cooperate with the shutdown method, executing code after the state change but also before the wait, probably the best way to implement it would be just to add an additional parameter to shutdown. "cancel" seems perfect, by default set to false, to not change the current behavior. Then maybe it would be a good idea to change the __exit__ method to set cancel to true if an exception is being handled, but I'm not sure about that. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XMDBSSMBXFJG2C3DXTNLTCNBGAUQOO2K/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
> gets one item from the queue, runs it, and then checks if the executor is > being shut down. That's exactly what I thought at first, but just after that the continue statement prevents that check, so all futures always get processed. Only when the sentinel is reached, which it's placed at the end, it actually does the check. > perhaps we can add some functionality where leftover work items are > explicitly cancelled? I think that would satisfy the OP's requirement. Yes that would be perfect, that way we could have a controlled but fast exit for certain scenarios, such as the ones listed in the first post. >From what I examined the procedure would be very similar to the one followed >when a initializer fails, both implementations clear all pending futures when >it happens. On ThreadPoolExecutor specifically, just calling a modified >version of _initializer_failed at the time discussed would seem to be >sufficient, changing the call Future.set_exception to Future.cancel. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RAFIAFGTPYTWQX2ZXLRK3IWYMYYMF7HD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
> Having a way to clear the queue and then shutdown once existing jobs are done > is a lot > more manageable. ... > So the only clean way to do this is cooperative: flush the queue, send some > kind of > message to all children telling them to finish as quickly as possible, then > wait for them > to finish. I was personally thinking of an implementation like that, cancel all still in pending and if wait is true the wait for the ones running, for both implementations. I didn't actually meant terminate literally, I just called it that as that's what multiprocessing.dummy.Pool.terminate (+ join after) does. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/M62FHIZGAPCUSULLVV63NEWKQ2HWH6OY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] concurrent.futures cancel pending at Executor.shutdown
Hi, I think it would be very helpful to have an additional argument (cancel for example) added to Executor.shutdown that cancels all pending futures submitted to the executor. Then context manager would gain the ability to abort all futures incase of a exception, additionally this would also implement the missing cuterpart of the multiprocessing module terminate, currently we only have close. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ZSUIFN5GTDO56H6LLOPXOLQK7EQQZJHJ/ Code of Conduct: http://python.org/psf/codeofconduct/
