thr, your solution of writing a Julia wrapper worked for me. thank you. i wouldn't have thought of that. how did you think to do that, by the way?
On Sunday, December 6, 2015 at 5:57:24 PM UTC-8, Yakir Gagnon wrote: > > OMG.... I went through so much round about ways to avoid that error... > This would be amazing if it works!!! Thanks! > > > Yakir Gagnon > The Queensland Brain Institute (Building #79) > The University of Queensland > Brisbane QLD 4072 > Australia > > cell +61 (0)424 393 332 > work +61 (0)733 654 089 > > On Mon, Dec 7, 2015 at 10:16 AM, thr <[email protected] <javascript:>> > wrote: > >> I think I found a way to use pycall in parallel: >> >> To get around the "cannot serialize pointer problem", I declared the >> corpus delicti everywhere and wrapped it inside a julia function that is >> also declared everywhere: >> It's also kind of clumsy and I'm not sure if it applies to your problem, >> but this worked for me: >> >> @everywhere using PyCall >> @everywhere @pyimport a_python_module >> >> @everywhere pyfun = a_python_function >> >> @everywhere function wrap_pyfun(x) >> return pyfun(x) >> end >> >> r1 = remotecall(2, wrap_pyfun, arg1) >> r2 = remotecall(3, wrap_pyfun, arg2) >> >> It also works if you have a python object and you need to call one of >> it's methods >> @everywhere pyobj = a_python_object >> >> @everywhere function wrap_pymethod(x) >> return pyobj[:method](x) >> end >> >> hope this helps >> >> On Saturday, October 31, 2015 at 2:43:19 AM UTC+1, Yakir Gagnon wrote: >> >>> OK, I found a horrible work around that might be good enough (for me): >>> Here is a mock python script: >>> >>> import math, time >>> x = math.sin(math.pi/4) >>> time.sleep(5) # mock local work >>> print(x) # spit it out to julia >>> >>> and here is the julia code that runs it: >>> >>> r = @spawn readall(`python example.py`) >>> sleep(2) # mock local work >>> wait(r) # wait on python >>> t = fetch(r) >>> x = parse(t) >>> x - sin(pi/4) # not zero but it works... >>> >>> This seems to work no matter what. It’s horrid, but better than nothing. >>> >>> On Friday, October 30, 2015 at 11:14:18 PM UTC+10, Matthew Pearce wrote: >>> >>> >>> So I got something working for my pylab example. >>> >>> julia> import PyCall >>> >>> julia> PyCall.@pyimport pylab >>> >>> julia> @everywhere import PyCall >>> >>> julia> @everywhere PyCall.@pyimport pylab >>> >>> julia> @everywhere A = pylab.cumsum(collect(1:10))*1. >>> >>> julia> fetch(@spawnat remotes[1] A) >>> 10-element Array{Float64,1}: >>> 1.0 >>> 3.0 >>> 6.0 >>> 10.0 >>> 15.0 >>> 21.0 >>> 28.0 >>> 36.0 >>> 45.0 >>> 55.0 >>> >>> >>> >>> >>> No luck with the math module I'm afraid. Two different types of errors >>> depending on style: >>> >>> julia> @spawnat remotes[1] PyCall.@pyimport math as pymath >>> RemoteRef{Channel{Any}}(2,1,305) >>> >>> julia> fetch(@spawnat remotes[1] (pymath.sin(pymath.pi / 4) - sin(pymath >>> .pi / 4)) ) >>> ERROR: On worker 2: >>> UndefVarError: pymath not defined >>> in anonymous at multi.jl:1330 >>> in anonymous at multi.jl:889 >>> in run_work_thunk at multi.jl:645 >>> in run_work_thunk at multi.jl:654 >>> in anonymous at task.jl:54 >>> in remotecall_fetch at multi.jl:731 >>> [inlined code] from multi.jl:368 >>> in call_on_owner at multi.jl:776 >>> in fetch at multi.jl:784 >>> >>> julia> @everywhere PyCall.@pyimport math as pymath >>> >>> julia> fetch(@spawnat remotes[1] (pymath.sin(pymath.pi / 4) - sin(pymath >>> .pi / 4)) ) >>> Worker 2 terminated.srun: error: mrc-bsu-tesla1: task 0: Exited with >>> exit code 1 >>> ERROR: ProcessExitedException() >>> in yieldto at ./task.jl:67 >>> in wait at ./task.jl:367 >>> in wait at ./task.jl:282 >>> in wait at ./channels.jl:97 >>> in take! at ./channels.jl:84 >>> in take! at ./multi.jl:792 >>> in remotecall_fetch at multi.jl:729 >>> [inlined code] from multi.jl:368 >>> in call_on_owner at multi.jl:776 >>> in fetch at multi.jl:784 >>> >>> >>> ERROR (unhandled task failure): EOFError: read end of file >>> >>> >>> >>> >>> >>> On Friday, October 30, 2015 at 1:28:21 AM UTC, Yakir Gagnon wrote: >>> >>> @Matthew: did you find a solution? >>> >>> On Tuesday, October 27, 2015 at 8:44:53 AM UTC+10, Yakir Gagnon wrote: >>> >>> Yea, right? So what’s the answer? How can we if at all do any PyCalls >>> parallely? >>> >>> On Monday, October 26, 2015 at 11:49:35 PM UTC+10, Matthew Pearce wrote: >>> >>> Thought I had an idea about this, I was wrong: >>> >>> ```julia >>> >>> julia> @everywhere using PyCall >>> >>> julia> @everywhere @pyimport pylab >>> >>> julia> remotecall_fetch(pylab.cumsum, 5, collect(1:10)) >>> ERROR: cannot serialize a pointer >>> [inlined code] from error.jl:21 >>> in serialize at serialize.jl:420 >>> [inlined code] from dict.jl:372 >>> in serialize at serialize.jl:428 >>> in serialize at serialize.jl:310 >>> in serialize at serialize.jl:420 (repeats 2 times) >>> in serialize at serialize.jl:302 >>> in serialize at serialize.jl:420 >>> [inlined code] from dict.jl:372 >>> in serialize at serialize.jl:428 >>> in serialize at serialize.jl:310 >>> in serialize at serialize.jl:420 (repeats 2 times) >>> in serialize at serialize.jl:302 >>> in serialize at serialize.jl:420 >>> [inlined code] from dict.jl:372 >>> in send_msg_ at multi.jl:222 >>> [inlined code] from multi.jl:177 >>> in remotecall_fetch at multi.jl:728 >>> [inlined code] from multi.jl:368 >>> in remotecall_fetch at multi.jl:734 >>> >>> julia> pylab.cumsum(collect(1:10)) >>> 10-element Array{Int64,1}: >>> 1 >>> 3 >>> 6 >>> 10 >>> 15 >>> 21 >>> 28 >>> 36 >>> 45 >>> 55 >>> >>> ``` >>> >>> >>> >>> ... >> >> >
