The example below demonstrates `remotecall_fetch` hanging when the remote
worker is under heavy CPU load.
It seems like the listener thread on the remote machine doesn't have high
enough priority to ensure that the remote call interrupts the busy while
loop.
This is of course disastrous since the remote call is an attempt to tell
the worker to take a break, and the worker is too busy to take heed.
*Is there anything that can be done in this case?*
This seems very bad for distributed technical computing.
Notes: Julia v0.4.5. Workers are on remote hosts. CentOS. I didn't
encounter this with all processes on one host (using a Windows 10 OS). This
only happened when I transferred code that works in my local environment to
an HPC cluster environment.
julia> workers() # These are all
on a remote host
12-element Array{Int64,1}:
14
15
16
17
18
19
20
21
22
23
24
25
julia> @everywhere module Test
stop_flag = false
function runner() global stop_flag; while(!stop_flag)
x=rand(5000,5000); end end
function stopper() global stop_flag; stop_flag = true; end
end
WARNING: replacing module Test
julia> r = remotecall(14, Test.runner)
RemoteRef{Channel{Any}}(14,1,148)
julia> r
RemoteRef{Channel{Any}}(14,1,148)
julia> r = remotecall_fetch(14, Test.stopper)
# Hangs here indefinitely
I attempted a fix, thinking that only 1 remote call might be allowed at a
time, but that proved a failure as well:
julia> @everywhere module Test
stop_flag = false
function runner() @async(begin global stop_flag; while(!stop_flag)
x=rand(5000,5000); end; end); println("runner running"); end
function stopper() global stop_flag; stop_flag = true; end
end
julia> remotecall_fetch( 2, Test.runner)
julia> remotecall_fetch(2, gethostname)
# Hangs indefinitely also
