On Mon, Mar 25, 2019 at 10:58 AM Jori Mäntysalo (TAU) <jori.mantys...@tuni.fi> wrote: > > Btw, what is most simple way in SageMath to run parallel independent jobs > without dependencies? > > For example, let G() be a generator outputting 1000 objects and let there > be four cpu cores available. Now it would be nice if we could just fork > four processes, each basically saying for example "Get an object o from > G(), compute f(o), and if it is 42, save o to the list F." This needs an > atomic way to call G(), and an atomic way to append a list. > > This seems so common form of computation that I suppose there is some nice > way for this.
For most cases you can try multiprocessing.Pool.map_async, and it works pretty nicely. It's biggest flaw for use in Sage is that it has a shortcoming where if an exception occurs in one of the worker processes, it only properly handles exceptions that inherit from Exception, and there are a few kinds of Python exceptions (e.g. KeyboardError) that do not. In conjunction with that, there's a bug (fixed in Python 3 though) that the map_async call would never return a result if one of the worker processes died unexpectedly (e.g. due to an unhandled segfault). But modulo those cases of crappy error handling, multiprocessing.Pool is exactly what you just described. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
