Re: [julia-users] Re: PyCall exception.jl's pycheckv - hidden exceptions or is it normal for that to take some time?

[Isaiah Norton]

>
> Aha - it makes complete sense that the Python workload would show up
> somewhere in the profiler. I suppose I wasn't expecting it in exception.jl,
> but if that is where the work happens that is fine.


It's an artifact of macro expansion not tracking where the expanded code
came from (fortunately there are a number of improvements in this area
coming in 0.5).

On Fri, Apr 15, 2016 at 11:47 AM, Tim Wheeler 
wrote:

> Aha - it makes complete sense that the Python workload would show up
> somewhere in the profiler. I suppose I wasn't expecting it in exception.jl,
> but if that is where the work happens that is fine.
> I'm copying the results from TF into my memory - if TF doesn't create new
> ones every time I can just retain handles to those but I haven't checked
> whether that is the case.
> Thank you for your advice!
>
>

[julia-users] Re: PyCall exception.jl's pycheckv - hidden exceptions or is it normal for that to take some time?

[Tim Wheeler]

Aha - it makes complete sense that the Python workload would show up 
somewhere in the profiler. I suppose I wasn't expecting it in exception.jl, 
but if that is where the work happens that is fine.
I'm copying the results from TF into my memory - if TF doesn't create new 
ones every time I can just retain handles to those but I haven't checked 
whether that is the case.
Thank you for your advice!

Re: [julia-users] Re: PyCall exception.jl's pycheckv - hidden exceptions or is it normal for that to take some time?

[Isaiah Norton]

>
> When passing arrays to Python, the PyCall default is already to use NumPy
> wrappers that pass the data without copying.


Yes, sorry for being unclear. My point was that this wrapper function ('
tfJuliaInterface.pass_image_to_ff') might not be taking advantage of the
existing NumPy-based buffer capability, since it seems to be spending so
much time copying.

On Fri, Apr 15, 2016 at 7:48 AM, Steven G. Johnson 
wrote:

>
>
> On Friday, April 15, 2016 at 12:14:53 AM UTC-4, Isaiah wrote:
>>
>> Your profiling result is not necessarily unreasonable. The listed line
>> number (exception.jl:78) is where the macro-wrapped code is actually
>> executed, and  "pass_image_to_ff" sounds like it could be expensive.
>>
>> Is the Julia-Pycall-TF wrapper copying data in that function? If so, then
>> it may be worth investigating the use of PyCall's buffer support instead,
>> to avoid copying (currently uses NumPy buffers underneath, see
>> https://github.com/stevengj/PyCall.jl/issues/38)
>>
>
> When passing arrays to Python, the PyCall default is already to use NumPy
> wrappers that pass the data without copying.
>

Re: [julia-users] Re: PyCall exception.jl's pycheckv - hidden exceptions or is it normal for that to take some time?

[Steven G. Johnson]

On Friday, April 15, 2016 at 12:14:53 AM UTC-4, Isaiah wrote:
>
> Your profiling result is not necessarily unreasonable. The listed line 
> number (exception.jl:78) is where the macro-wrapped code is actually 
> executed, and  "pass_image_to_ff" sounds like it could be expensive.
>
> Is the Julia-Pycall-TF wrapper copying data in that function? If so, then 
> it may be worth investigating the use of PyCall's buffer support instead, 
> to avoid copying (currently uses NumPy buffers underneath, see 
> https://github.com/stevengj/PyCall.jl/issues/38)
>

When passing arrays to Python, the PyCall default is already to use NumPy 
wrappers that pass the data without copying.

Re: [julia-users] Re: PyCall exception.jl's pycheckv - hidden exceptions or is it normal for that to take some time?

[Isaiah Norton]

Your profiling result is not necessarily unreasonable. The listed line
number (exception.jl:78) is where the macro-wrapped code is actually
executed, and  "pass_image_to_ff" sounds like it could be expensive.

Is the Julia-Pycall-TF wrapper copying data in that function? If so, then
it may be worth investigating the use of PyCall's buffer support instead,
to avoid copying (currently uses NumPy buffers underneath, see
https://github.com/stevengj/PyCall.jl/issues/38)

On Thu, Apr 14, 2016 at 2:40 PM, Tim Wheeler 
wrote:

> The relevant lines are:
>
> pycall(tfJuliaInterface.pass_image_to_ff, Void,
>weights, means, covs,
>model.sess, model.deepdrive, model.input_tensor)
> pyerr_check("pass image to ff")
>
> def pass_image_to_ff(weights, means, covs, sess, NN, image):
> feed = {NN.input_image: image, NN.is_training: False}
> w, mu, cov = sess.run([NN.weights_ff, NN.mu_ff, NN.cov_ff], feed)
> weights[:] = w
> means[:] = mu
> covs[:] = cov
>
>
>
> On Thursday, April 14, 2016 at 11:38:29 AM UTC-7, Tim Wheeler wrote:
>>
>> Hello Julia users,
>>
>> I am using PyCall to run some Python code. I did some profiling and found
>> that my time is dominated by the pycheckv macro.
>> My question is whether this is normal or if there are hidden exceptions
>> in my Python code that I need to worry about. pyerr_check is not
>> revealing any issues.
>>
>>
>> 
>>
>>
>>
>> thank you,
>> -Tim
>>
>

[julia-users] Re: PyCall exception.jl's pycheckv - hidden exceptions or is it normal for that to take some time?

[Tim Wheeler]

The relevant lines are:

pycall(tfJuliaInterface.pass_image_to_ff, Void,
   weights, means, covs,
   model.sess, model.deepdrive, model.input_tensor)
pyerr_check("pass image to ff")

def pass_image_to_ff(weights, means, covs, sess, NN, image):
feed = {NN.input_image: image, NN.is_training: False}
w, mu, cov = sess.run([NN.weights_ff, NN.mu_ff, NN.cov_ff], feed)
weights[:] = w
means[:] = mu
covs[:] = cov



On Thursday, April 14, 2016 at 11:38:29 AM UTC-7, Tim Wheeler wrote:
>
> Hello Julia users,
>
> I am using PyCall to run some Python code. I did some profiling and found 
> that my time is dominated by the pycheckv macro. 
> My question is whether this is normal or if there are hidden exceptions in 
> my Python code that I need to worry about. pyerr_check is not revealing 
> any issues.
>
>
> 
>
>
>
> thank you,
> -Tim
>