Thanks for all your hard work on the `cancel_futures` feature! As you said, there is a complexity cost (both in terms of the API and the implementation) whenever a new feature is added. The current ProcessPoolExecutor implementation, in particular, is complex enough that I can't easily easily reason about it's behavior and there are some long-standing bugs that have been difficult to resolve. I was actually hoping that we wouldn't touch the API for a while so I could do some stability work.
I'm -0 on the `cancel_on_error` feature but maybe it would help to hear about a concrete use case that can't be easily accomplished using the current API? Cheers, Brian On Mon, Feb 3, 2020 at 12:36 AM Kyle Stanley <aeros...@gmail.com> wrote: > Miguel Ángel Prosper wrote: > > Thank you so much for the work, I was very confused on how to even start > implementing it in the ProcessPoolExecutor, but you finished everything > super quick! > > No problem! The ProcessPoolExecutor implementation wasn't immediately > clear to me either, but after some experimentation (and some great > suggestions from Brian Quinlain to improve it) I was able to get it > functioning as intended. Thank you for the proposal, I think it will be a > good addition for the Executor API. > > Miguel Ángel Prosper wrote: > > I'm suppose that this might be better in another thread but... Could it > be possible to changing the context manager of the executor to cancel > futures on uncaught exceptions? > > Hmm, it should be possible. Do you specifically mean cancelling the > pending futures once a single one of the submitted functions raises an > exception, or cancelling the pending futures when the Executor itself > raises an exception (I.E. BrokenProcessPool)? I would assume the prior, > since that seems more useful to me. > > Miguel Ángel Prosper wrote: > > If not I could also be added as an optional parameter to the executor > constructor, "cancel_on_error" set to False to not changing anything. > > Assuming we did implement something like this, I'd say it should be done > as an optional parameter for the constructor (as stated above) rather than > as the default option when an exception occurs, because it would > significantly different from the current behavior. At the moment, the > executor doesn't mind when exceptions are raised from submitted functions; > it simply sets the exception to the future (via ``future.set_exception()``) > and moves on to the next. > > So, cancelling the pending futures after one raises an exception would be > a significant behavioral change; one that could easily result in breaking > existing code. As a result, I think it should be an optional parameter if > anything. > > As far as whether or not it should be implemented, I'm not 100% certain. > While I'm not against the idea of it, I think it would be reasonable to > wait until the new parameter is released in Python 3.9 and see what users > think of it prior to adding a similar parameter to the executor constructor > as well. The cost of adding a new feature demanded by users is relatively > minimal, but once it's added we're stuck with maintaining it. > > If the idea is approved, I'd be happy to help with implementing it. I just > want to make sure that we actually want it in the first place. > > What do you think, Guido? I've also CC'd Brian Quinlain and Antoine > Pitrou, in case they have any feedback but didn't see the original thread. > > On Mon, Feb 3, 2020 at 2:27 AM Miguel Ángel Prosper < > miguelangel.pros...@gmail.com> wrote: > >> Thank you so much for the work, I was very confused on how to even start >> implementing it in the ProcessPoolExecutor, but you finished everything >> super quick! >> >> I'm suppose that this might be better in another thread but... Could it >> be possible to changing the context manager of the executor to cancel >> futures on uncaught exceptions? If not I could also be added as an optional >> parameter to the executor constructor, "cancel_on_error" set to False to >> not changing anything. Personally I was thinking of using the executor like >> that; currently I have to either place everything in a try except block, >> handle it and then reraise it or subclass each executor. >> _______________________________________________ >> Python-ideas mailing list -- python-ideas@python.org >> To unsubscribe send an email to python-ideas-le...@python.org >> https://mail.python.org/mailman3/lists/python-ideas.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-ideas@python.org/message/R5VMWP75K35KF3DJN2F5MBZXD5UOQIV3/ >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5OK7BUEUACAHB7ZYHOPTXODVWCZWS5YB/ Code of Conduct: http://python.org/psf/codeofconduct/
