Steven, This https://github.com/JuliaLang/julia/pull/8507 should help when merged. A RemoteChannel can act as a work queue across workers.
Duane. Till support for channels are added to Base, you can check out and use https://github.com/amitmurthy/MessageUtils.jl On Thu, Oct 16, 2014 at 3:20 AM, Duane Johnson <[email protected]> wrote: > I'm trying to generalize my solution as a new `pmap` function: > > function pmap(fn::Function, producer::Task) > results = [] > np = nprocs() > @sync begin > for p = 1:np > if p != myid() || np == 1 > @async begin > for x in producer > r = remotecall_fetch(p, fn, x) > push!(results, r) > end > end > end > end > end > return results > end > > > However, I get the following error: > > ERROR: [] cannot grow. Instead, initialize the array with "T[]", where T > is the desired element type. > > I'm not sure how to pull the type of the producer out of `task`, since it > doesn't have a return type. Am I bumping in to this issue (github 1090) [1]? > > Is there a workaround? > > [1] https://github.com/JuliaLang/julia/issues/1090 > > On Wednesday, October 15, 2014 2:56:32 PM UTC-6, Duane Johnson wrote: >> >> The problem with the example in the manual is that you can't use a >> "producer" for the parallel map (pmap assumes the input has a fixed length). >> >> In my case, I would like to produce a list of files that may be very >> large (300k+) and I'd prefer not to wait until all filenames have been >> recursively discovered from disk before starting the work. >> >> On Wed, Oct 15, 2014 at 1:21 PM, Steven G. Johnson wrote: >> >>> There is an example in the manual of how to create a parallel work queue: >>> >>> http://docs.julialang.org/en/latest/manual/parallel- >>> computing/#scheduling >>> >>> (It would be nice to have cleaner built-in support for work queues and >>> work stealing [the advantage of work stealing is that it scales better >>> because there isn't a single queue as a bottleneck].) >>> >>> >>> >>
