Thanks for the reply and suggestion, Ralph.  I tried to get this working 
with semaphores/mutexes/locks/etc.  But I've not been having any luck.

Here's a simplified, incomplete version of what I'm trying to do.  I'm 
hoping that someone can offer a suggestion if they see some sample code.  

function localfunction()
    files = listfiles()
    locks = [Threads.SpinLock() for _ in files]
    ranges = getindexranges(length(files))

    pmap(pairs(ranges)) do rows_and_cols
        rows, cols = rows_and_cols
        workerfunction(files, locks, rows, cols)

function workerfunction(files, locks, rows, cols)
    data = kindofexpensive(...)
    pairs = pairs(rows, cols)
    @sync for idx in unique([rows; cols])
        @async begin
            updatefile(files[idx], data[idx])

This (obviously) does not work.  I think that the problem is that the locks 
are being copied when the function is spawned on each process.  I've tried 
wrapping the locks/semaphores in Futures/RemoteChannels, but that also 
hasn't worked for me.

I found that I could do the sort of coordination that I need by starting 
Tasks on the local process.  More specifically, each file would have an 
associated Task to handle the coordination between processes.  But this 
only worked for me in a simplified situation with the Tasks being declared 
globally.  When I tried to implement this coordination within localfunction 
above, I got an error (really a bunch of errors) that said that a running 
Task cannot be serialized.

Sorry for the long post, but I'm really hoping that someone can help me 
out.  I have a feeling that I'm missing something pretty simple.


On Tuesday, October 11, 2016 at 10:15:06 AM UTC-4, Ralph Smith wrote:
> You can do it with 2 (e.g. integer) channels per worker (requests and 
> replies) and a task for each pair in the main process. That's so ugly I'd 
> be tempted to write an
>  interface to named system semaphores. Or just use a separate file for 
> each worker.
> On Monday, October 10, 2016 at 11:09:39 AM UTC-4, Zachary Roth wrote:
>> Hi, everyone,
>> I'm trying to save to a single file from multiple worker processes, but 
>> don't know of a nice way to coordinate this.  When I don't coordinate, 
>> saving works fine much of the time.  But I sometimes get errors with 
>> reading/writing of files, which I'm assuming is happening because multiple 
>> processes are trying to use the same file simultaneously.
>> I tried to coordinate this with a queue/channel of `Condition`s managed 
>> by a task running in process 1, but this isn't working for me.  I've tried 
>> to simiplify this to track down the problem.  At least part of the issue 
>> seems to be writing to the channel from process 2.  Specifically, when I 
>> `put!` something onto a channel (or `push!` onto an array) from process 2, 
>> the channel/array is still empty back on process 1.  I feel like I'm 
>> missing something simple.  Is there an easier way to go about coordinating 
>> multiple processes that are trying to access the same file?  If not, does 
>> anyone have any tips?
>> Thanks for any help you can offer.
>> Cheers,
>> ---Zachary

Reply via email to