There are good synchronization primitives for Tasks, and a bit for threads, 
but not much for parallel processes. (One could use named system semaphores 
on Linux and Windows, but there's no Julia wrapper yet AFAIK.)
I also found management of parallel processes confusing, and good 
nontrivial examples are not obvious.  So here's my humble offering:

(I just happen to be working on some synchronization problems this month; 
if a real expert has a better solution, let's hear it.)

On Friday, October 14, 2016 at 3:19:25 PM UTC-4, Zachary Roth wrote:
> Thanks for the reply and suggestion, Ralph.  I tried to get this working 
> with semaphores/mutexes/locks/etc.  But I've not been having any luck.
> Here's a simplified, incomplete version of what I'm trying to do.  I'm 
> hoping that someone can offer a suggestion if they see some sample code.  
> function localfunction()
>     files = listfiles()
>     locks = [Threads.SpinLock() for _ in files]
>     ranges = getindexranges(length(files))
>     pmap(pairs(ranges)) do rows_and_cols
>         rows, cols = rows_and_cols
>         workerfunction(files, locks, rows, cols)
>     end
> end
> function workerfunction(files, locks, rows, cols)
>     data = kindofexpensive(...)
>     pairs = pairs(rows, cols)
>     @sync for idx in unique([rows; cols])
>         @async begin
>             lock(locks[idx])
>             updatefile(files[idx], data[idx])
>             unlock(locks[idx])
>         end
>     end
> end
> This (obviously) does not work.  I think that the problem is that the 
> locks are being copied when the function is spawned on each process.  I've 
> tried wrapping the locks/semaphores in Futures/RemoteChannels, but that 
> also hasn't worked for me.
> I found that I could do the sort of coordination that I need by starting 
> Tasks on the local process.  More specifically, each file would have an 
> associated Task to handle the coordination between processes.  But this 
> only worked for me in a simplified situation with the Tasks being declared 
> globally.  When I tried to implement this coordination within localfunction 
> above, I got an error (really a bunch of errors) that said that a running 
> Task cannot be serialized.
> Sorry for the long post, but I'm really hoping that someone can help me 
> out.  I have a feeling that I'm missing something pretty simple.
> ---Zachary
> On Tuesday, October 11, 2016 at 10:15:06 AM UTC-4, Ralph Smith wrote:
>> You can do it with 2 (e.g. integer) channels per worker (requests and 
>> replies) and a task for each pair in the main process. That's so ugly I'd 
>> be tempted to write an
>>  interface to named system semaphores. Or just use a separate file for 
>> each worker.
>> On Monday, October 10, 2016 at 11:09:39 AM UTC-4, Zachary Roth wrote:
>>> Hi, everyone,
>>> I'm trying to save to a single file from multiple worker processes, but 
>>> don't know of a nice way to coordinate this.  When I don't coordinate, 
>>> saving works fine much of the time.  But I sometimes get errors with 
>>> reading/writing of files, which I'm assuming is happening because multiple 
>>> processes are trying to use the same file simultaneously.
>>> I tried to coordinate this with a queue/channel of `Condition`s managed 
>>> by a task running in process 1, but this isn't working for me.  I've tried 
>>> to simiplify this to track down the problem.  At least part of the issue 
>>> seems to be writing to the channel from process 2.  Specifically, when I 
>>> `put!` something onto a channel (or `push!` onto an array) from process 2, 
>>> the channel/array is still empty back on process 1.  I feel like I'm 
>>> missing something simple.  Is there an easier way to go about coordinating 
>>> multiple processes that are trying to access the same file?  If not, does 
>>> anyone have any tips?
>>> Thanks for any help you can offer.
>>> Cheers,
>>> ---Zachary

Reply via email to