OMG.... I went through so much round about ways to avoid that error... This would be amazing if it works!!! Thanks!
Yakir Gagnon The Queensland Brain Institute (Building #79) The University of Queensland Brisbane QLD 4072 Australia cell +61 (0)424 393 332 work +61 (0)733 654 089 On Mon, Dec 7, 2015 at 10:16 AM, thr <[email protected]> wrote: > I think I found a way to use pycall in parallel: > > To get around the "cannot serialize pointer problem", I declared the > corpus delicti everywhere and wrapped it inside a julia function that is > also declared everywhere: > It's also kind of clumsy and I'm not sure if it applies to your problem, > but this worked for me: > > @everywhere using PyCall > @everywhere @pyimport a_python_module > > @everywhere pyfun = a_python_function > > @everywhere function wrap_pyfun(x) > return pyfun(x) > end > > r1 = remotecall(2, wrap_pyfun, arg1) > r2 = remotecall(3, wrap_pyfun, arg2) > > It also works if you have a python object and you need to call one of it's > methods > @everywhere pyobj = a_python_object > > @everywhere function wrap_pymethod(x) > return pyobj[:method](x) > end > > hope this helps > > On Saturday, October 31, 2015 at 2:43:19 AM UTC+1, Yakir Gagnon wrote: > >> OK, I found a horrible work around that might be good enough (for me): >> Here is a mock python script: >> >> import math, time >> x = math.sin(math.pi/4) >> time.sleep(5) # mock local work >> print(x) # spit it out to julia >> >> and here is the julia code that runs it: >> >> r = @spawn readall(`python example.py`) >> sleep(2) # mock local work >> wait(r) # wait on python >> t = fetch(r) >> x = parse(t) >> x - sin(pi/4) # not zero but it works... >> >> This seems to work no matter what. It’s horrid, but better than nothing. >> >> On Friday, October 30, 2015 at 11:14:18 PM UTC+10, Matthew Pearce wrote: >> >> >> So I got something working for my pylab example. >> >> julia> import PyCall >> >> julia> PyCall.@pyimport pylab >> >> julia> @everywhere import PyCall >> >> julia> @everywhere PyCall.@pyimport pylab >> >> julia> @everywhere A = pylab.cumsum(collect(1:10))*1. >> >> julia> fetch(@spawnat remotes[1] A) >> 10-element Array{Float64,1}: >> 1.0 >> 3.0 >> 6.0 >> 10.0 >> 15.0 >> 21.0 >> 28.0 >> 36.0 >> 45.0 >> 55.0 >> >> >> >> >> No luck with the math module I'm afraid. Two different types of errors >> depending on style: >> >> julia> @spawnat remotes[1] PyCall.@pyimport math as pymath >> RemoteRef{Channel{Any}}(2,1,305) >> >> julia> fetch(@spawnat remotes[1] (pymath.sin(pymath.pi / 4) - sin(pymath.pi >> / 4)) ) >> ERROR: On worker 2: >> UndefVarError: pymath not defined >> in anonymous at multi.jl:1330 >> in anonymous at multi.jl:889 >> in run_work_thunk at multi.jl:645 >> in run_work_thunk at multi.jl:654 >> in anonymous at task.jl:54 >> in remotecall_fetch at multi.jl:731 >> [inlined code] from multi.jl:368 >> in call_on_owner at multi.jl:776 >> in fetch at multi.jl:784 >> >> julia> @everywhere PyCall.@pyimport math as pymath >> >> julia> fetch(@spawnat remotes[1] (pymath.sin(pymath.pi / 4) - sin(pymath.pi >> / 4)) ) >> Worker 2 terminated.srun: error: mrc-bsu-tesla1: task 0: Exited with exit >> code 1 >> ERROR: ProcessExitedException() >> in yieldto at ./task.jl:67 >> in wait at ./task.jl:367 >> in wait at ./task.jl:282 >> in wait at ./channels.jl:97 >> in take! at ./channels.jl:84 >> in take! at ./multi.jl:792 >> in remotecall_fetch at multi.jl:729 >> [inlined code] from multi.jl:368 >> in call_on_owner at multi.jl:776 >> in fetch at multi.jl:784 >> >> >> ERROR (unhandled task failure): EOFError: read end of file >> >> >> >> >> >> On Friday, October 30, 2015 at 1:28:21 AM UTC, Yakir Gagnon wrote: >> >> @Matthew: did you find a solution? >> >> On Tuesday, October 27, 2015 at 8:44:53 AM UTC+10, Yakir Gagnon wrote: >> >> Yea, right? So what’s the answer? How can we if at all do any PyCalls >> parallely? >> >> On Monday, October 26, 2015 at 11:49:35 PM UTC+10, Matthew Pearce wrote: >> >> Thought I had an idea about this, I was wrong: >> >> ```julia >> >> julia> @everywhere using PyCall >> >> julia> @everywhere @pyimport pylab >> >> julia> remotecall_fetch(pylab.cumsum, 5, collect(1:10)) >> ERROR: cannot serialize a pointer >> [inlined code] from error.jl:21 >> in serialize at serialize.jl:420 >> [inlined code] from dict.jl:372 >> in serialize at serialize.jl:428 >> in serialize at serialize.jl:310 >> in serialize at serialize.jl:420 (repeats 2 times) >> in serialize at serialize.jl:302 >> in serialize at serialize.jl:420 >> [inlined code] from dict.jl:372 >> in serialize at serialize.jl:428 >> in serialize at serialize.jl:310 >> in serialize at serialize.jl:420 (repeats 2 times) >> in serialize at serialize.jl:302 >> in serialize at serialize.jl:420 >> [inlined code] from dict.jl:372 >> in send_msg_ at multi.jl:222 >> [inlined code] from multi.jl:177 >> in remotecall_fetch at multi.jl:728 >> [inlined code] from multi.jl:368 >> in remotecall_fetch at multi.jl:734 >> >> julia> pylab.cumsum(collect(1:10)) >> 10-element Array{Int64,1}: >> 1 >> 3 >> 6 >> 10 >> 15 >> 21 >> 28 >> 36 >> 45 >> 55 >> >> ``` >> >> >> >> ... > >
