I'm trying to generalize my solution as a new `pmap` function:
function pmap(fn::Function, producer::Task)
results = []
np = nprocs()
@sync begin
for p = 1:np
if p != myid() || np == 1
@async begin
for x in producer
r = remotecall_fetch(p, fn, x)
push!(results, r)
end
end
end
end
end
return results
end
However, I get the following error:
ERROR: [] cannot grow. Instead, initialize the array with "T[]", where T is
the desired element type.
I'm not sure how to pull the type of the producer out of `task`, since it
doesn't have a return type. Am I bumping in to this issue (github 1090) [1]?
Is there a workaround?
[1] https://github.com/JuliaLang/julia/issues/1090
On Wednesday, October 15, 2014 2:56:32 PM UTC-6, Duane Johnson wrote:
>
> The problem with the example in the manual is that you can't use a
> "producer" for the parallel map (pmap assumes the input has a fixed length).
>
> In my case, I would like to produce a list of files that may be very large
> (300k+) and I'd prefer not to wait until all filenames have been
> recursively discovered from disk before starting the work.
>
> On Wed, Oct 15, 2014 at 1:21 PM, Steven G. Johnson wrote:
>
>> There is an example in the manual of how to create a parallel work queue:
>>
>> http://docs.julialang.org/en/latest/manual/parallel-computing/#scheduling
>>
>> (It would be nice to have cleaner built-in support for work queues and
>> work stealing [the advantage of work stealing is that it scales better
>> because there isn't a single queue as a bottleneck].)
>>
>>
>>
>
