I often choose worker pools when there is working storage to be allocated as part of the work to be done. This way means that such data and processing structures are naturally built and torn down right there before and after the work-channel loop in the worker, with no repeated allocations during runtime, and no need for locked central data or data pools. It feels clean to me and stays clean even with work-stealing or worker-subdivision schemes.
Other ways make sense too, so not arguing for only one approach. Just saying that it works for me very well. Michael On Sat, Dec 28, 2019 at 8:17 AM Ian Lance Taylor <i...@golang.org> wrote: > On Sat, Dec 28, 2019 at 6:11 AM Robert Engels <reng...@ix.netcom.com> > wrote: > > > > Spinning up a Go routine when for each piece of work may be idiomatic > but it is far from the best practice for many situations - mainly because > of cpu cache invalidation. Most HPC systems create worker pools by type and > then isolate them to cpus/cores - something you can’t do in Go. > > > > I believe there are outstanding proposals for grouping routines and core > affinity. > > I think that today CPU cache invalidation is more or less independent > of spinning up separate goroutines. CPU cache invalidation is > something to consider for threads, but, as you say, goroutines have no > affinity to threads. Whether you use a worker pool or not, Go doesn't > currently give you any way to control how goroutines are assigned to > threads. So in general I don't see any reason to think that a worker > pool would be better or worse with regard to CPU cache invalidation. > > If Go introduces some way to associate goroutines with threads and > some way to associate threads with CPUs, that might well have an > effect on whether it is better to use a worker pool. > > Ian > > > > On Dec 28, 2019, at 7:02 AM, Agniva De Sarker < > agniva.quicksil...@gmail.com> wrote: > > > > > > > (the task was to limit the whole thing to about 10% of cores) > > > > I still don't think you needed a worker pool here. Like OP mentioned > above, you could just limit the number of goroutines executed to 10% of > total cores. > > > > > > On Saturday, 28 December 2019 18:02:08 UTC+5:30, Chris Burkert wrote: > >> > >> There are Pros and Cons for everything in life. Some time ago I wrote a > database tool which does something per table where the runtime largely > depends on the table size. I started with one goroutine per table because > it was easy. But that put a high load on the database (the task was to > limit the whole thing to about 10% of cores) with out of memory situations. > So I switched to a worker pool which solved that. However now the overall > runtime was unpredictable. When the large tables were handled at the end > their runtime defined the overall runtime. So I to feed the pool with large > tables first. This again lead to out of memory situations so I reordered > the tables such that large tables are mixed with smaller tables. > >> > >> Brian Candler <b.ca...@pobox.com> schrieb am Sa. 28. Dez. 2019 um > 11:09: > >>> > >>> On Friday, 27 December 2019 16:30:48 UTC, Bruno Albuquerque wrote: > >>>> > >>>> This might be useful too you, in any case: > >>>> > >>>> https://git.bug-br.org.br/bga/workerpool > >>>> > >>> > >>> I think the point from Bryan Mills' video is, "worker pool" is > something of an anti-pattern in go. goroutines are so cheap that you might > as well start a goroutine for each piece of work you have to do, and let it > terminate when that piece of work is done. > >>> > >>> Apart from the startup cost, the other reason for having a "worker > pool" is to limit the number of concurrent tasks being executed, and there > are better ways of doing that in go (also shown in the video). > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > an email to golan...@googlegroups.com. > >>> To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/f840beee-748f-42b6-809f-4c7505208aee%40googlegroups.com > . > > > > -- > > You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to golang-nuts+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/ecddafbc-eb73-45e1-8a5a-f738e88c6821%40googlegroups.com > . > > > > -- > > You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to golang-nuts+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/8C1153D5-25E7-4A06-8626-1FE61D9015D5%40ix.netcom.com > . > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CAOyqgcW9ORpo4dcUe6HYk1VS8Jp6J0EDpWAUXG4dHuYoUtbV7w%40mail.gmail.com > . > -- *Michael T. jonesmichael.jo...@gmail.com <michael.jo...@gmail.com>* -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CALoEmQyJKcP-21W7m60_DTN8tTntgTrX%3D%2BNqixudU1qEed%3D%2BJw%40mail.gmail.com.
