The code to sequence the results using a channel is not much more verbose. That not only avoids the library dependency, but also makes the peak memory consumption for the results O(runtime.NumCPU()), instead of O(N) with the number of tasks, and allows the output to be streamed instead of buffered to a slice.
https://play.golang.org/p/zkBjxlcvESe (In a more realistic situation I would probably use an errgroup.Group instead of a raw `go` statement, but the overall structure remains the same.) On Monday, January 28, 2019 at 3:49:34 AM UTC-5, rog wrote: > > FWIW I wrote *https://godoc.org/github.com/juju/utils/parallel#Run > <https://godoc.org/github.com/juju/utils/parallel#Run>* for exactly this > kind of use case. It doesn't explicitly mention the values in the API > because it's easier (and more type safe) to let the body of the closure > write directly to a result slice. > > Example: https://play.golang.org/p/ffj_Eb_0B2q > > > > On Fri, 25 Jan 2019, 10:48 pm <twp...@gmail.com <javascript:> wrote: > >> For what it's worth >> http://www.golangpatterns.info/concurrency/parallel-for-loop >> implements an order-preserving parallel map, but does not limit the >> number of workers. >> >> In my case, I want to limit the number of workers because I'm making a >> lot of system calls and don't want to overload the kernel. runtime.NumCPU() >> seems like a reasonable limit. >> >> >> >> On Friday, January 25, 2019 at 8:04:31 PM UTC+1, twp...@gmail.com wrote: >>> >>> Hi, >>> >>> I have a number of slow tasks that I want to run concurrently across >>> runtime.NumCPU() workers in a single process. The tasks have a specific >>> input order, but they are completely independent of each other and can >>> execute in any order. I would like to print the output of each task in the >>> same order as the input order of tasks. >>> >>> This can be implemented by including each task's index in the input >>> order as it is distributed via a channel to the workers, and the final >>> collection of results assembled using these task indexes before the results >>> are printed. >>> >>> Assumptions: >>> - Small number of tasks (~10,000 max), i.e. this easily fits in memory. >>> - Single Go process, i.e. I don't want/need a distributed system. >>> >>> This feels like it should be common problem and there's probably either >>> a library or a standard Go pattern out there which can do it. My web search >>> skills didn't find such a library though. Do you know of one? >>> >>> Cheers, >>> Tom >>> >>> >>> Background info to avoid the XY problem <http://xyproblem.info/>: this >>> is to make chezmoi <https://github.com/twpayne/chezmoi> run faster. I >>> want to run the doctor checks >>> <https://github.com/twpayne/chezmoi/blob/ed27b49f9ca4cd3662e6a59908dee24b0d295b79/cmd/doctor.go#L102-L163> >>> >>> (basically os.Exec'ing a whole load of binaries to get their versions) >>> concurrently in the short term. In the long term I want to make chezmoi's >>> apply concurrent, so it runs faster too. In the first case, the order >>> requirement is because I want all users to see the output in the same order >>> so that it's easy to compare. In the second case, the order requirement >>> comes because I need to ensure that parent directories are in the correct >>> state before checking their children. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
