Re: [theano-users] why does this gradient is invalid?

[佐藤優]

I understand.
I sincerely thank you.

Sato

2017年8月10日木曜日 7時39分21秒 UTC+9 nouiz:
>
> This is a bug in one Theano optimization: local_dimshuffle_subtensor
>
> Thanks for the report. I made an issue so that we don't forget it:
>
> https://github.com/Theano/Theano/issues/6288
>
> Frédéric
>
> On Wed, Aug 9, 2017 at 4:50 AM 佐藤優  wrote:
>
>> I wonder why bellow code is invalid..
>>
>> from numpy import *
>> import theano.tensor as T
>> x = T.dmatrix("x")
>> mx = x[...,None,:]
>> a = T.ones((1,3))
>> T.grad(mx[...,0].dot(a).sum(), a).eval({x:ones((5,10)).astype(float32)})
>>
>> bellow error is emerged.
>>
>> ---ValueError
>> Traceback (most recent call 
>> last)/home/yu/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py
>>  in __call__(self, *args, **kwargs)883 outputs =\--> 884 
>> self.fn() if output_subset is None else\885 
>> self.fn(output_subset=output_subset)
>> ValueError: Shape mismatch: A.shape[1] != x.shape[0]
>>
>> During handling of the above exception, another exception occurred:
>> ValueErrorTraceback (most recent call 
>> last) in ()  3 mx = x[...,None,:] 
>>  4 a = T.ones((1,3))> 5 T.grad(mx[...,0].dot(a).sum(), 
>> a).eval({x:ones((5,10)).astype(float32)})
>> /home/yu/anaconda3/lib/python3.5/site-packages/theano/gof/graph.py in 
>> eval(self, inputs_to_values)517 args = [inputs_to_values[param] 
>> for param in inputs]518 --> 519 rval = 
>> self._fn_cache[inputs](*args)520 521 return rval
>> /home/yu/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py
>>  in __call__(self, *args, **kwargs)896 
>> node=self.fn.nodes[self.fn.position_of_error],897 
>> thunk=thunk,--> 898 storage_map=getattr(self.fn, 
>> 'storage_map', None))899 else:900 # 
>> old-style linkers raise their own exceptions
>> /home/yu/anaconda3/lib/python3.5/site-packages/theano/gof/link.py in 
>> raise_with_op(node, thunk, exc_info, storage_map)323 # extra 
>> long error message in that case.324 pass--> 325 
>> reraise(exc_type, exc_value, exc_trace)326 327 
>> /home/yu/anaconda3/lib/python3.5/site-packages/six.py in reraise(tp, value, 
>> tb)683 value = tp()684 if value.__traceback__ is 
>> not tb:--> 685 raise value.with_traceback(tb)686 
>> raise value687 
>> /home/yu/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py
>>  in __call__(self, *args, **kwargs)882 try:883 
>> outputs =\--> 884 self.fn() if output_subset is None else\   
>>  885 self.fn(output_subset=output_subset)886 
>> except Exception:
>> ValueError: Shape mismatch: A.shape[1] != x.shape[0]
>> Apply node that caused the error: 
>> CGemv{inplace}(AllocEmpty{dtype='float64'}.0, TensorConstant{1.0}, 
>> InplaceDimShuffle{1,0}.0, Rebroadcast{0}.0, TensorConstant{0.0})
>> Toposort index: 7
>> Inputs types: [TensorType(float64, vector), TensorType(float64, scalar), 
>> TensorType(float64, matrix), TensorType(float64, vector), 
>> TensorType(float64, scalar)]
>> Inputs shapes: [(3,), (), (3, 5), (1,), ()]
>> Inputs strides: [(8,), (), (8, 24), (80,), ()]
>> Inputs values: [array([  0.e+000,   4.94065646e-324,   
>> 9.88131292e-324]), array(1.0), 'not shown', array([ 1.]), array(0.0)]
>> Inputs type_num: [12, 12, 12, 12, 12]
>> Outputs clients: [[InplaceDimShuffle{x,0}(CGemv{inplace}.0)]]
>>
>> Debugprint of the apply node: 
>> CGemv{inplace} [id A]  ''   
>>  |AllocEmpty{dtype='float64'} [id B]  ''   
>>  | |TensorConstant{3} [id C] 
>>  |TensorConstant{1.0} [id D] 
>>  |InplaceDimShuffle{1,0} [id E]  ''   
>>  | |Alloc [id F]  ''   
>>  |   |TensorConstant{(1, 1) of 1.0} [id G] > True))>
>>  |   |Shape_i{0} [id H]  ''   
>>  |   | |x [id I] 
>>  |   |TensorConstant{3} [id C] 
>>  |Rebroadcast{0} [id J]  ''   
>>  | |Subtensor{int8, ::, int64} [id K]  ''   
>>  |   |InplaceDimShuffle{0,x,1} [id L] > False))> ''   
>>  |   | |x [id I] 
>>  |   |Constant{0} [id M] 
>>  |   |Constant{0} [id N] 
>>  |TensorConstant{0.0} [id O] 
>>
>> Storage map footprint:
>>  - x, Input, Shape: (5, 10), ElemSize: 8 Byte(s), TotalSize: 400 Byte(s)
>>  - 

Re: [theano-users] why does this gradient is invalid?

[Frédéric Bastien]

This is a bug in one Theano optimization: local_dimshuffle_subtensor

Thanks for the report. I made an issue so that we don't forget it:

https://github.com/Theano/Theano/issues/6288

Frédéric

On Wed, Aug 9, 2017 at 4:50 AM 佐藤優  wrote:

> I wonder why bellow code is invalid..
>
> from numpy import *
> import theano.tensor as T
> x = T.dmatrix("x")
> mx = x[...,None,:]
> a = T.ones((1,3))
> T.grad(mx[...,0].dot(a).sum(), a).eval({x:ones((5,10)).astype(float32)})
>
> bellow error is emerged.
>
> ---ValueError
> Traceback (most recent call 
> last)/home/yu/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py
>  in __call__(self, *args, **kwargs)883 outputs =\--> 884  
>self.fn() if output_subset is None else\885 
> self.fn(output_subset=output_subset)
> ValueError: Shape mismatch: A.shape[1] != x.shape[0]
>
> During handling of the above exception, another exception occurred:
> ValueErrorTraceback (most recent call 
> last) in ()  3 mx = x[...,None,:]  
> 4 a = T.ones((1,3))> 5 T.grad(mx[...,0].dot(a).sum(), 
> a).eval({x:ones((5,10)).astype(float32)})
> /home/yu/anaconda3/lib/python3.5/site-packages/theano/gof/graph.py in 
> eval(self, inputs_to_values)517 args = [inputs_to_values[param] 
> for param in inputs]518 --> 519 rval = 
> self._fn_cache[inputs](*args)520 521 return rval
> /home/yu/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py
>  in __call__(self, *args, **kwargs)896 
> node=self.fn.nodes[self.fn.position_of_error],897 
> thunk=thunk,--> 898 storage_map=getattr(self.fn, 
> 'storage_map', None))899 else:900 # 
> old-style linkers raise their own exceptions
> /home/yu/anaconda3/lib/python3.5/site-packages/theano/gof/link.py in 
> raise_with_op(node, thunk, exc_info, storage_map)323 # extra long 
> error message in that case.324 pass--> 325 reraise(exc_type, 
> exc_value, exc_trace)326 327
> /home/yu/anaconda3/lib/python3.5/site-packages/six.py in reraise(tp, value, 
> tb)683 value = tp()684 if value.__traceback__ is 
> not tb:--> 685 raise value.with_traceback(tb)686 
> raise value687
> /home/yu/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py
>  in __call__(self, *args, **kwargs)882 try:883 
> outputs =\--> 884 self.fn() if output_subset is None else\
> 885 self.fn(output_subset=output_subset)886 
> except Exception:
> ValueError: Shape mismatch: A.shape[1] != x.shape[0]
> Apply node that caused the error: 
> CGemv{inplace}(AllocEmpty{dtype='float64'}.0, TensorConstant{1.0}, 
> InplaceDimShuffle{1,0}.0, Rebroadcast{0}.0, TensorConstant{0.0})
> Toposort index: 7
> Inputs types: [TensorType(float64, vector), TensorType(float64, scalar), 
> TensorType(float64, matrix), TensorType(float64, vector), TensorType(float64, 
> scalar)]
> Inputs shapes: [(3,), (), (3, 5), (1,), ()]
> Inputs strides: [(8,), (), (8, 24), (80,), ()]
> Inputs values: [array([  0.e+000,   4.94065646e-324,   
> 9.88131292e-324]), array(1.0), 'not shown', array([ 1.]), array(0.0)]
> Inputs type_num: [12, 12, 12, 12, 12]
> Outputs clients: [[InplaceDimShuffle{x,0}(CGemv{inplace}.0)]]
>
> Debugprint of the apply node:
> CGemv{inplace} [id A]  ''
>  |AllocEmpty{dtype='float64'} [id B]  ''
>  | |TensorConstant{3} [id C] 
>  |TensorConstant{1.0} [id D] 
>  |InplaceDimShuffle{1,0} [id E]  ''
>  | |Alloc [id F]  ''
>  |   |TensorConstant{(1, 1) of 1.0} [id G] 
>  |   |Shape_i{0} [id H]  ''
>  |   | |x [id I] 
>  |   |TensorConstant{3} [id C] 
>  |Rebroadcast{0} [id J]  ''
>  | |Subtensor{int8, ::, int64} [id K]  ''
>  |   |InplaceDimShuffle{0,x,1} [id L]  False))> ''
>  |   | |x [id I] 
>  |   |Constant{0} [id M] 
>  |   |Constant{0} [id N] 
>  |TensorConstant{0.0} [id O] 
>
> Storage map footprint:
>  - x, Input, Shape: (5, 10), ElemSize: 8 Byte(s), TotalSize: 400 Byte(s)
>  - InplaceDimShuffle{0,x,1}.0, Shape: (5, 1, 10), ElemSize: 8 Byte(s), 
> TotalSize: 400 Byte(s)
>  - Alloc.0, Shape: (5, 3), ElemSize: 8 Byte(s), TotalSize: 120 Byte(s)
>  - InplaceDimShuffle{1,0}.0, Shape: (3, 5), ElemSize: 8 Byte(s), 

Re: [theano-users] Split Op (OpFromGraph) to save intermediate results for grad

[Frédéric Bastien]

Sorry, but I'm not able to answer this grad question. Hopefully someone
else that better understand that part can answer.

Fred

On Mon, Jul 31, 2017 at 9:43 AM  wrote:

> I am trying to build an Op with a custom/optimized gradient formula. To
> override the automatic differenciation, I'm trying to use OpFromGraph.
> The gradient formula can reuse intermediate results from the feed forward
> pass, so I have tried to split the Op in two: Op1 computes the intermediate
> and final result and gives all of it to Op2, Op2 forwards the final result
> and takes care of the gradient computation given all the necessary values.
>
> Note that the gradient of the loss wrt the intermediate results is never
> needed.
>
> Below is a what I believe to be a minimal working example of my problem,
> it exhibits a strange conversion error related to the gradient computation
> with the intermediate values. Please take note of the presence of an
> integral variable.
>
> import numpy as np
> import theano.tensor as T
> import theano
>
>
> def make_ops():
> x = T.vector()
> m = T.bvector()
>
> r = m.sum().astype('floatX')  # intermediate value
> z = x * m / r  # final result
>
>
> def grad_op1(inputs, output_gradients):
> return [
> output_gradients[0],  # gradient computation delegated to op2
> T.DisconnectedType()()  # variable has integral type
> # T.zeros_like(inputs[1])
> ]
>
>
> op1 = theano.OpFromGraph(
> inputs=[x, m],
> outputs=[z, m, r],
> grad_overrides=grad_op1,
> inline=True,
> name="op1")
>
>
> z = T.vector()
> r_forwarded = T.scalar()
>
> def grad_op2(inputs, output_gradients):
> _, m_, r_ = inputs
> dm_ = theano.gradient.DisconnectedType()(name="dm_")
> # I think the error could be around here
> <<--
> # dr_ = theano.gradient.DisconnectedType()(name="dr_")
> dr_ = T.zeros_like(r_)
> return [m_ / r_, dm_, dr_]
>
> op2 = theano.OpFromGraph(
> inputs=[z, m, r_forwarded],
> outputs=[z],  # Op 2 forwards the precomputed output
> grad_overrides=grad_op2,
> inline=True,
> name="op2")
>
> return op1, op2
>
>
> def main():
> op1, op2 = make_ops()
> x = T.vector(name="x")
> m = T.bvector(name="m")
> z_intermediate, m_forwarded, r = op1(x, m)
> z = op2(z_intermediate, m, r)
>
> g = theano.grad(T.sum(z), wrt=x)
> print(g.eval({x: np.array([1., .3, .0, .2], dtype=np.float32),
>   m: np.array([1, 0, 1, 1], dtype=np.int8)}))
>
>
> if __name__ == "__main__":
> main()
>
> (Note: I had tried to hijack my previous question thread with this problem
> but it went unnoticed, sorry for double posting)
>
> Thank you
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to theano-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Why is this GpuFromHost call generated?

[Frédéric Bastien]

Hi,

do you use float? I was meaning float32. The old back-end only suport
float32. So if you use float64 or int32, nothing will compute on the GPU.

The new back-end support many dtypes including float64 and int*. So it
should work better.

Note, if you do operation between float32 and int32, the result is float64.
This is the normal c/numpy casting rules. float32 and int16 return float32.
So if you end up with float64, it is frequently that case.
Fred

On Wed, Aug 9, 2017 at 2:48 PM Haining Yu  wrote:

> Thank you Fred.
>
> Yes I am using device=gpu0. I will switch to the new backend and test
> again.
>
> On float64, do you mean int64? If yes, am puzzled by that too. In my code
> I never explicit cast to int64. Instead I use tensor.ivector() to index
> matrices and cast them explicitly into int32. For example:
>
> x = T.ivector()
>
> z = T.cast(y, dtype='int32')
>
> Do you think these things cause the problem?
>
> Thank you,
> Haining
>
> Haining Yu on Gmail
>
> On Wed, Aug 9, 2017 at 2:36 PM, Frédéric Bastien <
> frederic.bast...@gmail.com> wrote:
>
>> My guess is that you use the old GPU backend. Can you confirm you use the
>> Theano flag device=gpu? And that also you have float64 in the graph. The
>> old backend don't support them. I suggest that you install the just
>> released 0.10 beta and that you use the new backend with device=cuda.
>>
>> Also,you can use the flag warn_float64=pdb to find where you have them
>> and make sore they are float32. This will be faster.
>>
>> Fred
>>
>> Le lun. 31 juil. 2017 14:42, Haining Yu  a écrit :
>>
>>> Hi,
>>>
>>> I am running a RNN/GRU model for a fairly large dataset with the goal
>>> of sequence prediction. When I profile my code, I found one GpuFromHost
>>> takes ~30% of computation time. See part of profiling results below:
>>>
>>> <% time><#call>  
>>>  
>>>   30.2%73.0% 462.776s   3.71e-01s   1248   221
>>>   GpuFromHost(Subtensor{:int64:}.0)
>>> input 0: dtype=float32, shape=(512, 1024, 2048), strides=(-4096, 4,
>>> 2097152)
>>> output 0: dtype=float32, shape=(512, 1024, 2048), strides=(2097152,
>>> 2048, 1)
>>>
>>> theano.printing.debugprint shows that the call is generated in gradient
>>> calculation; see snippet below. There is also a HostFromGpu a couple of
>>> layers below.
>>>
>>>  | | | | |GpuFromHost [id FN] ''   221
>>>  | | | |   |Subtensor{:int64:} [id FO] ''   220
>>>  | | | | |Subtensor{::int64} [id FP] ''   219
>>>  | | | | | |InplaceDimShuffle{1,2,0} [id FQ] ''   218
>>>  | | | | | | |Reshape{3} [id FR] ''   217
>>>  | | | | | |   |CrossentropyCategorical1HotGrad [id FS] ''   216
>>>  | | | | | |   | |Elemwise{Second}[(0, 0)] [id FT] ''   215
>>>  | | | | | |   | | |CrossentropyCategorical1Hot [id FU] ''   209
>>>  | | | | | |   | | | |HostFromGpu [id FV] ''   206
>>>
>>> I have heard about the cost of using GpuFromHost (and its counterpart
>>> HostFromGpu) and had moved almost all data to GPU (via shared
>>> variables). So I don't understand why the call is needed. In particular I
>>> don't understand:
>>>
>>> 1. If all my data are on GPU and theano is optimized for GPU, why is the
>>> GpuFromHost even generated?
>>> 2. Is the call generated because the memory is too large? The call tries
>>> to move 512 x 1024 x 2048 x 4 = 4.2GB memory. But my Tesla K80 should have
>>> 12GB memory thus the need to move seems remote on the surface. Overall
>>> memory consumption seems OK under profiling.
>>> 3. Does the call have anything to do with CrossentropyCategorical1Hot? I
>>> assume CrossentropyCategorical1Hot  has been optimized for GPU. But the
>>> code shows that a HostFromGPU is called before CrossentropyCategorical1Hot
>>> is applied. I am not sure if CrossentropyCategorical1Hot has any memory
>>> requirement (e.g., c-contiguous).
>>> 4. Should I try any GPU assertion to debug the root cause of the problem?
>>>
>>> Any hint is appreciated.
>>>
>>> Thank you,
>>> Haining
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "theano-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to theano-users+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>>
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "theano-users" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/theano-users/CjR0L_KroOU/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> theano-users+unsubscr...@googlegroups.com.
>>
>
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to 

Re: [theano-users] Error while compiling two theano functions

[Frédéric Bastien]

note, I made an issue about this:

https://github.com/Theano/Theano/issues/6287

Fred

On Mon, Jul 3, 2017 at 7:51 AM Frédéric Bastien 
wrote:

> This is still experimental and we don't have to to work on it now.
>
> For multiple GPU, you should do data parallelism. The is 3 framework that
> can help you, theano-mpi, platoon and synkronous.
>
> Fred
>
> Le sam. 1 juil. 2017 16:33, Ramana Subramanyam  a
> écrit :
>
>> Hi,
>> This error that I reported was solved using the
>> flag optimizer_excluding=fusion. However, when I try to use multiple GPUs,
>> I get this error
>>
>> ERROR (theano.gof.opt): Optimization failure due to:
>> LocalOptGroup(local_abstractconv_cudnn,local_abstractconv_gw_cudnn,local_abstractconv_gi_cudnn,local_abstractconv_gemm,local_abstractconv3d_gemm,local_abstractconv_gradweights_gemm,local_abstractconv3d_gradweights_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gradinputs_gemm)
>> ERROR (theano.gof.opt): node: AbstractConv2d{convdim=2, border_mode=(4,
>> 3), subsample=(1, 1), filter_flip=False, imshp=(None, None, None, None),
>> kshp=(None, None, None, None), filter_dilation=(1, 1)}(X,
>> CIFAR10.pixelCNN.pxCNN.vstack1.filter)
>> ERROR (theano.gof.opt): TRACEBACK:
>> ERROR (theano.gof.opt): Traceback (most recent call last):
>>   File
>> "/home/akshat/anaconda2/envs/ramana-test/lib/python2.7/site-packages/theano/gof/opt.py",
>> line 1982, in process_node
>> replacements = lopt.transform(node)
>>   File
>> "/home/akshat/anaconda2/envs/ramana-test/lib/python2.7/site-packages/theano/gof/opt.py",
>> line 1335, in transform
>> new_repl = opt.transform(node)
>>   File
>> "/home/akshat/anaconda2/envs/ramana-test/lib/python2.7/site-packages/theano/gpuarray/dnn.py",
>> line 2816, in local_abstractconv_cudnn
>> ctx = infer_context_name(*node.inputs)
>>   File
>> "/home/akshat/anaconda2/envs/ramana-test/lib/python2.7/site-packages/theano/gpuarray/basic_ops.py",
>> line 122, in infer_context_name
>> raise ValueError("Could not infer context from inputs")
>> ValueError: Could not infer context from inputs
>>
>> I used these THEANO_FLAGS
>> , contexts=dev0->cuda1;dev1->cuda3,floatX=float32,optimizer_excluding=fusion.
>> The same flags works well with the import and the sample code on this
>> page
>> . This
>> is my first time using multiple GPUs, apologise if I have made some trivial
>> mistake
>>
>> Ramana
>>
>>
>> On Tuesday, June 27, 2017 at 11:50:12 PM UTC+5:30, Ramana Subramanyam
>> wrote:
>>>
>>> Hi Fred,
>>> Since there wasn't any \n in the output, it was all in the same line.
>>> You have to scroll towards your left/right on this link
>>> . I am pasting a smaller copy of that below,
>>>
>>>
>>> (Composite{Switch((LT(i0, i1), i1, i0)}(Composite{Switch(GE(i0, i1), i1,
>>> i0)}(i0, i1), i2), i3), Composite{Switch(LT(i0, i1), i1,
>>> i0)}(Composite{Switch(GE(i0, i1), i1, i0)}(i0, i1), i2), i3) + i4)}(i8,
>>> Composite{((i0 + i1) - i2)}(i2, Composite{Switch(LT(Composite{Switch(GE(i0,
>>> i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{((i0 + i1) -
>>> i2)}(i0, i1, i2), i3, i4), i5), i3), i3, Composite{Switch(GE(i0, i1), i1,
>>> i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{((i0 + i1) - i2)}(i0,
>>> i1, i2), i3, i4), i5))}(i1, Composite{Switch(LT(Composite{Switch(GE(i0,
>>> i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{((i0 + i1) -
>>> i2)}(i0, i1, i2), i3, i4), i5), i3), i3, Composite{Switch(GE(i0, i1), i1,
>>> i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{((i0 + i1) - i2)}(i0,
>>> i1, i2), i3, i4), i5))}(i2, i3, i4, i5, i6, Composite{((i0 + i1) - i2)}(i7,
>>> i3, i4)), Composite{(Switch(LT(Composite{Switch(LT(i0, i1), i1,
>>> i0)}(Composite{Switch(GE(i0, i1), i1, i0)}(i0, i1), i2), i3),
>>> Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(GE(i0, i1), i1,
>>> i0)}(i0, i1), i2), i3) + i4)}(i8, Composite{((i0 + i1) - i2)}(i7, i3, i4),
>>> i5, Composite{Switch(LT(Composite{Switch(GE(i0, i1), i1,
>>> i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{((i0 + i1) - i2)}(i0,
>>> i1, i2), i3, i4), i5), i3), i3, Composite{Switch(GE(i0, i1), i1,
>>> i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{((i0 + i1) - i2)}(i0,
>>> i1, i2), i3, i4), i5))}(i2, i3, i4, i5, i6, Composite{((i0 + i1) - i2)}(i7,
>>> i3, i4)), i9), i5, i6, Composite{((i0 + i1) - i2)}(i2,
>>> Composite{Switch(LT(Composite{Switch(GE(i0, i1), i1,
>>> i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{((i0 + i1) - i2)}(i0,
>>> i1, i2), i3, i4), i5), i3), i3, Composite{Switch(GE(i0, i1), i1,
>>> i0)}(Composite{Switch(LT(i0, i1), i2, i0)}
>>>
>>> On Tuesday, June 27, 2017 at 11:44:11 PM UTC+5:30, nouiz wrote:

 The output you gave don't show the infinit loop printed. Can you give
 me longer output?

 On Tue, Jun 27, 2017 at 2:08 PM Ramana Subramanyam 
 wrote:

Re: [theano-users] How to build different average pooling operation I'll call it local average pooling ?

[Jesse Livezey]

I think this idea would be something like

y = [1, 2, 3, 0]

y_current_avgpool = (1 + 2 + 3 + 0) / 4 

y_new_avgpool = (1 + 2 + 3) / 3 

I'm not sure that there is a simple way to do this currently. You could do 
sum pooling first, then compute the divisors by looking at the number of 
non-zero elements using this
http://deeplearning.net/software/theano/library/tensor/nnet/neighbours.html#theano.tensor.nnet.neighbours.images2neibs
and T.switch

On Wednesday, August 9, 2017 at 11:36:29 AM UTC-7, nouiz wrote:
>
> I don't understand the problem with using normal operation. Can you give 
> this code? I don't see more problem with that implementation vs a normal 
> average pooling.
>
> Le mar. 8 août 2017 07:36, Feras Almasri  
> a écrit :
>
>> I want to have an node that take the average of the only activated points 
>> in the last feature map. what I mean by activated points is any pixel 
>> higher than zero. instead of taking the global average of the full feature 
>> map I'd rather take it of the only activated pixels.
>> If I just do this in normal operation then gradient descent will be 
>> discontinued in a way that location for back prorogation are not there. 
>>
>> Any hint or advice would be appreciated, thank you. 
>>
>> -- 
>>
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "theano-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to theano-users...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Why is this GpuFromHost call generated?

[Haining Yu]

Thank you Fred.

Yes I am using device=gpu0. I will switch to the new backend and test again.

On float64, do you mean int64? If yes, am puzzled by that too. In my code I
never explicit cast to int64. Instead I use tensor.ivector() to index
matrices and cast them explicitly into int32. For example:

x = T.ivector()

z = T.cast(y, dtype='int32')

Do you think these things cause the problem?

Thank you,
Haining

Haining Yu on Gmail

On Wed, Aug 9, 2017 at 2:36 PM, Frédéric Bastien  wrote:

> My guess is that you use the old GPU backend. Can you confirm you use the
> Theano flag device=gpu? And that also you have float64 in the graph. The
> old backend don't support them. I suggest that you install the just
> released 0.10 beta and that you use the new backend with device=cuda.
>
> Also,you can use the flag warn_float64=pdb to find where you have them and
> make sore they are float32. This will be faster.
>
> Fred
>
> Le lun. 31 juil. 2017 14:42, Haining Yu  a écrit :
>
>> Hi,
>>
>> I am running a RNN/GRU model for a fairly large dataset with the goal of
>> sequence prediction. When I profile my code, I found one GpuFromHost takes
>> ~30% of computation time. See part of profiling results below:
>>
>> <% time><#call>  
>>  
>>   30.2%73.0% 462.776s   3.71e-01s   1248   221
>>   GpuFromHost(Subtensor{:int64:}.0)
>> input 0: dtype=float32, shape=(512, 1024, 2048), strides=(-4096, 4,
>> 2097152)
>> output 0: dtype=float32, shape=(512, 1024, 2048), strides=(2097152,
>> 2048, 1)
>>
>> theano.printing.debugprint shows that the call is generated in gradient
>> calculation; see snippet below. There is also a HostFromGpu a couple of
>> layers below.
>>
>>  | | | | |GpuFromHost [id FN] ''   221
>>  | | | |   |Subtensor{:int64:} [id FO] ''   220
>>  | | | | |Subtensor{::int64} [id FP] ''   219
>>  | | | | | |InplaceDimShuffle{1,2,0} [id FQ] ''   218
>>  | | | | | | |Reshape{3} [id FR] ''   217
>>  | | | | | |   |CrossentropyCategorical1HotGrad [id FS] ''   216
>>  | | | | | |   | |Elemwise{Second}[(0, 0)] [id FT] ''   215
>>  | | | | | |   | | |CrossentropyCategorical1Hot [id FU] ''   209
>>  | | | | | |   | | | |HostFromGpu [id FV] ''   206
>>
>> I have heard about the cost of using GpuFromHost (and its counterpart
>> HostFromGpu) and had moved almost all data to GPU (via shared
>> variables). So I don't understand why the call is needed. In particular I
>> don't understand:
>>
>> 1. If all my data are on GPU and theano is optimized for GPU, why is the
>> GpuFromHost even generated?
>> 2. Is the call generated because the memory is too large? The call tries
>> to move 512 x 1024 x 2048 x 4 = 4.2GB memory. But my Tesla K80 should have
>> 12GB memory thus the need to move seems remote on the surface. Overall
>> memory consumption seems OK under profiling.
>> 3. Does the call have anything to do with CrossentropyCategorical1Hot? I
>> assume CrossentropyCategorical1Hot  has been optimized for GPU. But the
>> code shows that a HostFromGPU is called before CrossentropyCategorical1Hot
>> is applied. I am not sure if CrossentropyCategorical1Hot has any memory
>> requirement (e.g., c-contiguous).
>> 4. Should I try any GPU assertion to debug the root cause of the problem?
>>
>> Any hint is appreciated.
>>
>> Thank you,
>> Haining
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "theano-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to theano-users+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "theano-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/theano-users/CjR0L_KroOU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> theano-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Grouped Convolution Error

[Frédéric Bastien]

Their have been a fix in Theano. Can you update and try again?

Le lun. 24 juil. 2017 19:56, Michael Klachko  a
écrit :

> I'm trying the new grouped convolutions feature in the latest Theano
> version, so I ran a simple convnet with CIFAR-10: 32x32 RGB input images
> (batch size = 128), and the first convolutional layer has 9 feature maps. I
> want to have 3 feature maps per color, so if I understand it correctly, I
> should use num_groups=3 argument in conv2d op.
>
> Again: I want the first conv. layer to process input images with 3 filters
> per color, so that each color channel is connected to 3 feature maps.
> Filters are 8x8 with stride 8 (non-overlapping) so the output feature maps
> should be 4x4 pixels.
>
> After adding the num_groups arg I got the following error:
>
> ValueError: images and kernel must have the same stack size
> Apply node that caused the error: GpuDnnConv{algo='time_on_shape_change',
> inplace=True, num_groups=3}(GpuContiguous.0, GpuContiguous.0,
> GpuAllocEmpty{dtype='float32', context_name=None}.0, GpuDnnConvDesc{
> border_mode='valid', subsample=(8, 8), dilation=(1, 1), conv_mode='conv',
> precision='float32'}.0, Constant{1.0}, Constant{0.0})
> Toposort index: 62
> Inputs types: [GpuArrayType(float32, 4D), GpuArrayType(float32
> , 4D), GpuArrayType(float32, 4D),  at 0x7fa3900bc910>, Scalar(float32), Scalar(float32)]
> Inputs shapes: [(128, 3, 32, 32), (9, 3, 8, 8), (128, 9, 4, 4), 'No
> shapes', (), ()]
> Inputs strides: [(12288, 4096, 128, 4), (768, 256, 32, 4), (576, 64, 16, 4
> ), 'No strides', (), ()]
> Inputs values: ['not shown', 'not shown', 'not shown',  NULL at 0x7fa372027f30>, 1.0, 0.0]
> Outputs clients: [[GpuElemwise{Add}[(0, 0)](GpuDnnConv{algo=
> 'time_on_shape_change', inplace=True, num_groups=3}.0,
> InplaceGpuDimShuffle{x,0,x,x}.0)]]
>
>
>
> Thanks,
> Michael
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to theano-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Error with theano. This was working fine earlier

[Frédéric Bastien]

You changed something in your installation. Try to delete your Theano
cache. If that don't fix it try to remove all your Python. You probably
have mixed Python in your environment.

Le mer. 19 juil. 2017 10:37, SUNITHA  a écrit :

> Dear All,
>
> This is the error message I get:
>
> *C:\Users\Sunitha\Anaconda2\libs/python27.lib: error adding symbols: File
> in wrong format collect2.exe: error: ld returned 1 exit status*
>
> 1 #include 
> 2 #include "theano_mod_helper.h"
> 3 #include "structmember.h"
> 4 #include 
> 5
> 6 #if PY_VERSION_HEX >= 0x0300
> 7 #include "numpy/npy_3kcompat.h"
> 8 #define PyCObject_AsVoidPtr  NpyCapsule_AsVoidPtr
> 9 #define PyCObject_GetDesc  NpyCapsule_GetDesc
> 00010 #define PyCObject_Check NpyCapsule_Check
> 00011 #endif
> 00012
>
> Please help me fix this issue.
>
> Regards,
> Sunitha
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to theano-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Theano ConftestImportFailure error after installing

[Frédéric Bastien]

We don't use py.test, but nosetests.

Fred

Le mar. 8 août 2017 12:12, Sara Saeed  a écrit :

>
> I am new to Ubuntu and I tried to install Theano using Anaconda.
>
> After tracking some other errors and solving them. I am stuck with this
> error, which I don't understand when running py.test.
>
> Can anyone help me to fix this.
>
> Thank you
>
>
>
>
>
>
>
>
> ===1 error in 6.61 seconds
> 
> (theanoenv) sara@sara-ubunto:-$ AC
> (theano_env) sara@sara-ubunto:-$ AC
> (theano_env) sara@sara-ubunto:-$ AC
> (theano_env) sara@sara-ubunto:-$ AC
> (theano_env) sara@sara-ubunto:-$ AC
> (theano_env) sara@sara-ubunto:-$ theano-cache clear
> (theano_env) sara@sara-ubunto:~$ PY.tast
> test session stats
> =
> platform linux2 -- Python 2.7.13, pytest-3.1.,r py-1.4 4, pluggy-0.4.0
> rootdir: /home/sara, inifile:
> collected 0 items / 1 errors
>
>
> ===ERRORS
> ===
> __EROR collecting
> __
> anaconda3/envs/theano_env/lib/python2.7/site-package/py/_path/common.py:
> 372: In
> visit
>  for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
> anaconda3/envs/theano_env/lib/python2.7/site-packages/py/_path/common.py:
> 421: in
> gen
>  for p in self.gen(subdir):
> anaconda3/envs/theano_env/lib/python2.7/site-packages/py/_path/common.py:
> 421: in
> gen
>  for p in self.gen(subdir):
> anaconda3/envs/theano_env/lib/python2.7/site-packages/py/_path/common.py:
> 421: in
> gen
>  for p in self.gen(subdir):
> anaconda3/envs/theano_env/lib/python2.7/site-packages/py/_path/common.py :
> 421: in
> gen
>  for p in self.gen(subdir):
> anaconda3/envs/theano_env/lib/python2.7/site-packages/py/_path/common.py:
> 411: in
> gen
>  if p.check(dir=1) and (rec is None or rec(p))])
> anaconda3/envs/theano_env/lib/python2.7/site-packages/_pytest/main.py:686:
> in
> recurse
>  ihook = self.gethookproxy(path)
> anaconda3/envs/theano_env/lib/python2.7/site-packages/_pytest/main.py:590:
> in ge
> thookproxy
>  my_conftestodules = pm._getconftestmodules(fspath)
> anaconda3/envs/theano_env/lib/python2.7/site-packages/_pytest/config.py:
> 350: in
> _getconftestmodules
>  mod = self.importconftest(conftestpath)
> anaconda3/envs/theano_env/lib/python2.7/site-packages/_pytest/config.py:
> 375: in
> _importconftest
>  raise ConftestImportFailure(conftestpath, sys.exc_info())
>  onftestImportrailure: ImportflismatchErroR('pandas.conftest', '/home/sara/an
>
> aconda3/envs/theano_env/lib/python2.7/site-packages/pandas/conftest.py',
> local('
> /home/sara/anaconda3/lib/python3.6/site-packages/pandas/conftest.py'))
> !! Interrupted: 1 errors during collection
> !
> ==1error in 11.59 seconds
> 
> (theano_env) sara@sara-ubunto:~$
>
>
>
>
>
> 
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to theano-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Theano flatten doesn't give expected output with arguments ndim

[Lalit Pradhan]

I have variables 'a'  and 'b' which are  
'theano.sandbox.cuda.var.CudaNdarraySharedVariable'. 
I am passing an array of shape ((1,128,300,300)) into 'a'
I am passing an array of shape ((1,1,300,300)) into 'b'
c = a*b . type(c) = 'theano.tensor.var.TensorVariable' of shape 
((1,128,300,300))
c = c.flatten(ndim=3), I expect c to be of shape ((1,128,9)) but it 
returns ((1,1,1152)) instead. Flattening 'a' and 'b'  gives expected 
results but not 'c'. 
Can someone explain the reason for it?

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Split Op (OpFromGraph) to save intermediate results for grad

[nicolas . granger . m]

"forward the precomputed output" means that Op1 already computed the final 
output, therefore Op2 just has to behaves as identity in the forward pass 

The intermediate value is already an output of Op1 as shown in the example 
code, sorry if that wasn't clear.

Nicolas

Le mardi 8 août 2017 20:56:12 UTC+2, nouiz a écrit :
>
> I don't understand what you mean by "forward the precomputed output"
>
> What I would recommand is to make 1 op for the forward. The intermediate 
> value that can be reused for the gradient, make them output. Don't use them 
> in the forward, but you can reuse them your grad override.
>
> Frédéric
>
> On Mon, Jul 31, 2017 at 9:43 AM  
> wrote:
>
>> I am trying to build an Op with a custom/optimized gradient formula. To 
>> override the automatic differenciation, I'm trying to use OpFromGraph. 
>> The gradient formula can reuse intermediate results from the feed forward 
>> pass, so I have tried to split the Op in two: Op1 computes the intermediate 
>> and final result and gives all of it to Op2, Op2 forwards the final result 
>> and takes care of the gradient computation given all the necessary values.
>>
>> Note that the gradient of the loss wrt the intermediate results is never 
>> needed.
>>
>> Below is a what I believe to be a minimal working example of my problem, 
>> it exhibits a strange conversion error related to the gradient computation 
>> with the intermediate values. Please take note of the presence of an 
>> integral variable.
>>
>> import numpy as np
>> import theano.tensor as T
>> import theano
>>
>>
>> def make_ops():
>> x = T.vector()
>> m = T.bvector()
>>
>> r = m.sum().astype('floatX')  # intermediate value
>> z = x * m / r  # final result
>>
>>
>> def grad_op1(inputs, output_gradients):
>> return [
>> output_gradients[0],  # gradient computation delegated to op2
>> T.DisconnectedType()()  # variable has integral type
>> # T.zeros_like(inputs[1])
>> ]
>>
>>
>> op1 = theano.OpFromGraph(
>> inputs=[x, m],
>> outputs=[z, m, r],
>> grad_overrides=grad_op1,
>> inline=True,
>> name="op1")
>>
>>
>> z = T.vector()
>> r_forwarded = T.scalar()
>>
>> def grad_op2(inputs, output_gradients):
>> _, m_, r_ = inputs
>> dm_ = theano.gradient.DisconnectedType()(name="dm_")
>> # I think the error could be around here 
>> <<--
>> # dr_ = theano.gradient.DisconnectedType()(name="dr_")
>> dr_ = T.zeros_like(r_)
>> return [m_ / r_, dm_, dr_]
>>
>> op2 = theano.OpFromGraph(
>> inputs=[z, m, r_forwarded],
>> outputs=[z],  # Op 2 forwards the precomputed output
>> grad_overrides=grad_op2,
>> inline=True,
>> name="op2")
>>
>> return op1, op2
>>
>>
>> def main():
>> op1, op2 = make_ops()
>> x = T.vector(name="x")
>> m = T.bvector(name="m")
>> z_intermediate, m_forwarded, r = op1(x, m)
>> z = op2(z_intermediate, m, r)
>>
>> g = theano.grad(T.sum(z), wrt=x)
>> print(g.eval({x: np.array([1., .3, .0, .2], dtype=np.float32),
>>   m: np.array([1, 0, 1, 1], dtype=np.int8)}))
>>
>>
>> if __name__ == "__main__":
>> main()
>>
>> (Note: I had tried to hijack my previous question thread with this 
>> problem but it went unnoticed, sorry for double posting)
>>
>> Thank you
>>
>> -- 
>>
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "theano-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to theano-users...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] why does this gradient is invalid?

[佐藤優]

I wonder why bellow code is invalid..

from numpy import *
import theano.tensor as T
x = T.dmatrix("x")
mx = x[...,None,:]
a = T.ones((1,3))
T.grad(mx[...,0].dot(a).sum(), a).eval({x:ones((5,10)).astype(float32)})

bellow error is emerged.

---ValueError
Traceback (most recent call 
last)/home/yu/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py
 in __call__(self, *args, **kwargs)883 outputs =\--> 884
 self.fn() if output_subset is None else\885 
self.fn(output_subset=output_subset)
ValueError: Shape mismatch: A.shape[1] != x.shape[0]

During handling of the above exception, another exception occurred:
ValueErrorTraceback (most recent call 
last) in ()  3 mx = x[...,None,:]
  4 a = T.ones((1,3))> 5 T.grad(mx[...,0].dot(a).sum(), 
a).eval({x:ones((5,10)).astype(float32)})
/home/yu/anaconda3/lib/python3.5/site-packages/theano/gof/graph.py in 
eval(self, inputs_to_values)517 args = [inputs_to_values[param] for 
param in inputs]518 --> 519 rval = self._fn_cache[inputs](*args)
520 521 return rval
/home/yu/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py
 in __call__(self, *args, **kwargs)896 
node=self.fn.nodes[self.fn.position_of_error],897 
thunk=thunk,--> 898 storage_map=getattr(self.fn, 
'storage_map', None))899 else:900 # 
old-style linkers raise their own exceptions
/home/yu/anaconda3/lib/python3.5/site-packages/theano/gof/link.py in 
raise_with_op(node, thunk, exc_info, storage_map)323 # extra long 
error message in that case.324 pass--> 325 reraise(exc_type, 
exc_value, exc_trace)326 327 
/home/yu/anaconda3/lib/python3.5/site-packages/six.py in reraise(tp, value, tb) 
   683 value = tp()684 if value.__traceback__ is not 
tb:--> 685 raise value.with_traceback(tb)686 raise 
value687 
/home/yu/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py
 in __call__(self, *args, **kwargs)882 try:883 
outputs =\--> 884 self.fn() if output_subset is None else\
885 self.fn(output_subset=output_subset)886 except 
Exception:
ValueError: Shape mismatch: A.shape[1] != x.shape[0]
Apply node that caused the error: CGemv{inplace}(AllocEmpty{dtype='float64'}.0, 
TensorConstant{1.0}, InplaceDimShuffle{1,0}.0, Rebroadcast{0}.0, 
TensorConstant{0.0})
Toposort index: 7
Inputs types: [TensorType(float64, vector), TensorType(float64, scalar), 
TensorType(float64, matrix), TensorType(float64, vector), TensorType(float64, 
scalar)]
Inputs shapes: [(3,), (), (3, 5), (1,), ()]
Inputs strides: [(8,), (), (8, 24), (80,), ()]
Inputs values: [array([  0.e+000,   4.94065646e-324,   
9.88131292e-324]), array(1.0), 'not shown', array([ 1.]), array(0.0)]
Inputs type_num: [12, 12, 12, 12, 12]
Outputs clients: [[InplaceDimShuffle{x,0}(CGemv{inplace}.0)]]

Debugprint of the apply node: 
CGemv{inplace} [id A]  ''   
 |AllocEmpty{dtype='float64'} [id B]  ''   
 | |TensorConstant{3} [id C] 
 |TensorConstant{1.0} [id D] 
 |InplaceDimShuffle{1,0} [id E]  ''   
 | |Alloc [id F]  ''   
 |   |TensorConstant{(1, 1) of 1.0} [id G] 
 |   |Shape_i{0} [id H]  ''   
 |   | |x [id I] 
 |   |TensorConstant{3} [id C] 
 |Rebroadcast{0} [id J]  ''   
 | |Subtensor{int8, ::, int64} [id K]  ''   
 |   |InplaceDimShuffle{0,x,1} [id L]  ''   
 |   | |x [id I] 
 |   |Constant{0} [id M] 
 |   |Constant{0} [id N] 
 |TensorConstant{0.0} [id O] 

Storage map footprint:
 - x, Input, Shape: (5, 10), ElemSize: 8 Byte(s), TotalSize: 400 Byte(s)
 - InplaceDimShuffle{0,x,1}.0, Shape: (5, 1, 10), ElemSize: 8 Byte(s), 
TotalSize: 400 Byte(s)
 - Alloc.0, Shape: (5, 3), ElemSize: 8 Byte(s), TotalSize: 120 Byte(s)
 - InplaceDimShuffle{1,0}.0, Shape: (3, 5), ElemSize: 8 Byte(s), TotalSize: 120 
Byte(s)
 - AllocEmpty{dtype='float64'}.0, Shape: (3,), ElemSize: 8 Byte(s), TotalSize: 
24 Byte(s)
 - Subtensor{int8, ::, int64}.0, Shape: (1,), ElemSize: 8 Byte(s), TotalSize: 8 
Byte(s)
 - Shape_i{0}.0, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
 - TensorConstant{1.0}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
 - TensorConstant{0.0}, Shape: (), ElemSize: 8