Thanks for the reply and suggestion, Ralph. I tried to get this working with semaphores/mutexes/locks/etc. But I've not been having any luck.
Here's a simplified, incomplete version of what I'm trying to do. I'm hoping that someone can offer a suggestion if they see some sample code. function localfunction() files = listfiles() locks = [Threads.SpinLock() for _ in files] ranges = getindexranges(length(files)) pmap(pairs(ranges)) do rows_and_cols rows, cols = rows_and_cols workerfunction(files, locks, rows, cols) end end function workerfunction(files, locks, rows, cols) data = kindofexpensive(...) pairs = pairs(rows, cols) @sync for idx in unique([rows; cols]) @async begin lock(locks[idx]) updatefile(files[idx], data[idx]) unlock(locks[idx]) end end end This (obviously) does not work. I think that the problem is that the locks are being copied when the function is spawned on each process. I've tried wrapping the locks/semaphores in Futures/RemoteChannels, but that also hasn't worked for me. I found that I could do the sort of coordination that I need by starting Tasks on the local process. More specifically, each file would have an associated Task to handle the coordination between processes. But this only worked for me in a simplified situation with the Tasks being declared globally. When I tried to implement this coordination within localfunction above, I got an error (really a bunch of errors) that said that a running Task cannot be serialized. Sorry for the long post, but I'm really hoping that someone can help me out. I have a feeling that I'm missing something pretty simple. ---Zachary On Tuesday, October 11, 2016 at 10:15:06 AM UTC-4, Ralph Smith wrote: > > You can do it with 2 (e.g. integer) channels per worker (requests and > replies) and a task for each pair in the main process. That's so ugly I'd > be tempted to write an > interface to named system semaphores. Or just use a separate file for > each worker. > > On Monday, October 10, 2016 at 11:09:39 AM UTC-4, Zachary Roth wrote: >> >> Hi, everyone, >> >> I'm trying to save to a single file from multiple worker processes, but >> don't know of a nice way to coordinate this. When I don't coordinate, >> saving works fine much of the time. But I sometimes get errors with >> reading/writing of files, which I'm assuming is happening because multiple >> processes are trying to use the same file simultaneously. >> >> I tried to coordinate this with a queue/channel of `Condition`s managed >> by a task running in process 1, but this isn't working for me. I've tried >> to simiplify this to track down the problem. At least part of the issue >> seems to be writing to the channel from process 2. Specifically, when I >> `put!` something onto a channel (or `push!` onto an array) from process 2, >> the channel/array is still empty back on process 1. I feel like I'm >> missing something simple. Is there an easier way to go about coordinating >> multiple processes that are trying to access the same file? If not, does >> anyone have any tips? >> >> Thanks for any help you can offer. >> >> Cheers, >> ---Zachary >> >