I think I found a way to use pycall in parallel:
To get around the "cannot serialize pointer problem", I declared the corpus
delicti everywhere and wrapped it inside a julia function that is also
declared everywhere:
It's also kind of clumsy and I'm not sure if it applies to your problem,
but this worked for me:
@everywhere using PyCall
@everywhere @pyimport a_python_module
@everywhere pyfun = a_python_function
@everywhere function wrap_pyfun(x)
return pyfun(x)
end
r1 = remotecall(2, wrap_pyfun, arg1)
r2 = remotecall(3, wrap_pyfun, arg2)
It also works if you have a python object and you need to call one of it's
methods
@everywhere pyobj = a_python_object
@everywhere function wrap_pymethod(x)
return pyobj[:method](x)
end
hope this helps
On Saturday, October 31, 2015 at 2:43:19 AM UTC+1, Yakir Gagnon wrote:
>
> OK, I found a horrible work around that might be good enough (for me):
> Here is a mock python script:
>
> import math, time
> x = math.sin(math.pi/4)
> time.sleep(5) # mock local work
> print(x) # spit it out to julia
>
> and here is the julia code that runs it:
>
> r = @spawn readall(`python example.py`)
> sleep(2) # mock local work
> wait(r) # wait on python
> t = fetch(r)
> x = parse(t)
> x - sin(pi/4) # not zero but it works...
>
> This seems to work no matter what. It’s horrid, but better than nothing.
>
> On Friday, October 30, 2015 at 11:14:18 PM UTC+10, Matthew Pearce wrote:
>
>
> So I got something working for my pylab example.
>
> julia> import PyCall
>
> julia> PyCall.@pyimport pylab
>
> julia> @everywhere import PyCall
>
> julia> @everywhere PyCall.@pyimport pylab
>
> julia> @everywhere A = pylab.cumsum(collect(1:10))*1.
>
> julia> fetch(@spawnat remotes[1] A)
> 10-element Array{Float64,1}:
> 1.0
> 3.0
> 6.0
> 10.0
> 15.0
> 21.0
> 28.0
> 36.0
> 45.0
> 55.0
>
>
>
>
> No luck with the math module I'm afraid. Two different types of errors
> depending on style:
>
> julia> @spawnat remotes[1] PyCall.@pyimport math as pymath
> RemoteRef{Channel{Any}}(2,1,305)
>
> julia> fetch(@spawnat remotes[1] (pymath.sin(pymath.pi / 4) - sin(pymath.pi
> / 4)) )
> ERROR: On worker 2:
> UndefVarError: pymath not defined
> in anonymous at multi.jl:1330
> in anonymous at multi.jl:889
> in run_work_thunk at multi.jl:645
> in run_work_thunk at multi.jl:654
> in anonymous at task.jl:54
> in remotecall_fetch at multi.jl:731
> [inlined code] from multi.jl:368
> in call_on_owner at multi.jl:776
> in fetch at multi.jl:784
>
> julia> @everywhere PyCall.@pyimport math as pymath
>
> julia> fetch(@spawnat remotes[1] (pymath.sin(pymath.pi / 4) - sin(pymath.pi
> / 4)) )
> Worker 2 terminated.srun: error: mrc-bsu-tesla1: task 0: Exited with exit
> code 1
> ERROR: ProcessExitedException()
> in yieldto at ./task.jl:67
> in wait at ./task.jl:367
> in wait at ./task.jl:282
> in wait at ./channels.jl:97
> in take! at ./channels.jl:84
> in take! at ./multi.jl:792
> in remotecall_fetch at multi.jl:729
> [inlined code] from multi.jl:368
> in call_on_owner at multi.jl:776
> in fetch at multi.jl:784
>
>
> ERROR (unhandled task failure): EOFError: read end of file
>
>
>
>
>
> On Friday, October 30, 2015 at 1:28:21 AM UTC, Yakir Gagnon wrote:
>
> @Matthew: did you find a solution?
>
> On Tuesday, October 27, 2015 at 8:44:53 AM UTC+10, Yakir Gagnon wrote:
>
> Yea, right? So what’s the answer? How can we if at all do any PyCalls
> parallely?
>
> On Monday, October 26, 2015 at 11:49:35 PM UTC+10, Matthew Pearce wrote:
>
> Thought I had an idea about this, I was wrong:
>
> ```julia
>
> julia> @everywhere using PyCall
>
> julia> @everywhere @pyimport pylab
>
> julia> remotecall_fetch(pylab.cumsum, 5, collect(1:10))
> ERROR: cannot serialize a pointer
> [inlined code] from error.jl:21
> in serialize at serialize.jl:420
> [inlined code] from dict.jl:372
> in serialize at serialize.jl:428
> in serialize at serialize.jl:310
> in serialize at serialize.jl:420 (repeats 2 times)
> in serialize at serialize.jl:302
> in serialize at serialize.jl:420
> [inlined code] from dict.jl:372
> in serialize at serialize.jl:428
> in serialize at serialize.jl:310
> in serialize at serialize.jl:420 (repeats 2 times)
> in serialize at serialize.jl:302
> in serialize at serialize.jl:420
> [inlined code] from dict.jl:372
> in send_msg_ at multi.jl:222
> [inlined code] from multi.jl:177
> in remotecall_fetch at multi.jl:728
> [inlined code] from multi.jl:368
> in remotecall_fetch at multi.jl:734
>
> julia> pylab.cumsum(collect(1:10))
> 10-element Array{Int64,1}:
> 1
> 3
> 6
> 10
> 15
> 21
> 28
> 36
> 45
> 55
>
> ```
>
>
>
> ...
