If I understood your goal correctly, there is a tiny python function I have
been pasting into projects for ages:
def chunks(l, n):
"""Yield successive n-sized chunks from l.
From
http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python/312464#312464
"""
for i in xrange(0, len(l), n):
yield l[i:i + n]
I converted it to Julia, and since it uses Tasks, I think it will work with
Julia's async support (I haven't tried Julia's parallel tools much yet):
function chunks(a, n::Int)
collect(Task() do
for i in 1:n:length(a) produce(a[i:min(length(a), i + n - 1)]) end
end)
end
println(chunks([1:20], 5))
println(chunks([1:20], 3))
for chunk in chunks([1:20], 7)
println(chunk)
end
@parallel for chunk in chunks([1:20], 7)
println(chunk)
end
On Wednesday, June 11, 2014 12:58:57 PM UTC-4, krishna mohan wrote:
>
> Dear All
> Thanks for pointing out.
>
> I am trying to break the L array into chunks of 5 elements and pass it
> @async to dowork() and wait with @sync, once done process the next chunk
> from the array.
>
> (ofcourse I will do addprocs() for using multiple processors)
>
> Regards,
> Krishna
>
>
>
> On Wed, Jun 11, 2014 at 9:30 PM, <[email protected] <javascript:>> wrote:
>
>> Many things are not right.
>>
>>
>>> L = 1:1000
>>> stride = 5
>>>
>>> function dowork(i)
>>> if i < 1000*: **not in python*
>>> sleep(5)
>>> else
>>> break *you are assuming your code will be inlined, but it won't*
>>> end *an extra end is missing here*
>>>
>>
>> That specific piece seems weird, I'm not sure this is legal but at the
>> very least, this is confusing:
>>
>>> i = 1
>>> @sync @parallel for i in i:i+stride
>>> dowork(i)
>>> i += 1
>>> end
>>>
>>
>> I would propose:
>> j=1
>> ...
>> @parallel for i... j:j+stride
>> ...
>> j += stride + 1
>> ...
>> to make things clear again. I'm not sure you'll get what you want though.
>>
>> If you want dowork to break the loop, you should have it return
>> something, like a boolean.
>> if dowork(i) break end
>>
>> I've not spent much time yet at coding in parallel however, I'm really
>> not sure @sync behaves as you expect
>> <http://julia.readthedocs.org/en/latest/manual/parallel-computing/#synchronization-with-remote-references>.
>>
>> From what I've understood, @sync acts like a barrier: the @sync block will
>> surrender the control to the main task once every @async jobs are run (but
>> I may well be totally wrong). In your case, if the job for i = 1500
>> finishes before i = 1499, your loop will be broken without dowork(1499)
>> being calculated. This is the ugliest kind of bugs one can encounter: the
>> ones that occur at random.
>>
>
>
