Re: [theano-users] How to build different average pooling operation I'll call it local average pooling ?

[Jesse Livezey]

I think this idea would be something like

y = [1, 2, 3, 0]

y_current_avgpool = (1 + 2 + 3 + 0) / 4 

y_new_avgpool = (1 + 2 + 3) / 3 

I'm not sure that there is a simple way to do this currently. You could do 
sum pooling first, then compute the divisors by looking at the number of 
non-zero elements using this
http://deeplearning.net/software/theano/library/tensor/nnet/neighbours.html#theano.tensor.nnet.neighbours.images2neibs
and T.switch

On Wednesday, August 9, 2017 at 11:36:29 AM UTC-7, nouiz wrote:
>
> I don't understand the problem with using normal operation. Can you give 
> this code? I don't see more problem with that implementation vs a normal 
> average pooling.
>
> Le mar. 8 août 2017 07:36, Feras Almasri  
> a écrit :
>
>> I want to have an node that take the average of the only activated points 
>> in the last feature map. what I mean by activated points is any pixel 
>> higher than zero. instead of taking the global average of the full feature 
>> map I'd rather take it of the only activated pixels.
>> If I just do this in normal operation then gradient descent will be 
>> discontinued in a way that location for back prorogation are not there. 
>>
>> Any hint or advice would be appreciated, thank you. 
>>
>> -- 
>>
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "theano-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to theano-users...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Theano GPU results not reproducible

[Jesse Livezey]

Some operations on GPU are not deterministic. I think some convolution 
operations and also reduction operations are two examples. See this thread 
for more info
https://groups.google.com/forum/#!searchin/theano-users/atomic$20add%7Csort:relevance/theano-users/g-BF6zwMirM/ojWzbUBPBwAJ

On Tuesday, July 18, 2017 at 1:20:41 PM UTC-7, Wenpeng Yin wrote:
>
> Hi guys,
>
> I have a long-term problem when running theano code in GPU: even I use two 
> command windows to run the same program (on the same GPU or different 
> GPUs), they show different performances. It's hard to say the difference is 
> small or big, depending on the task.  This makes difficult to judge a 
> program modification is better or worse.
>
> I can not find the problem, as I notice that I always use the same random 
> seed, for example "rng = numpy.random.RandomState(23455)",  whenever I 
> create parameters, so they are expected to repeat the process, right? 
>
> The only thing I can think about is that GPU uses 32 bits, not 64, this 
> will lose precision?
>
> Thanks for any hints.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Error performing 2D convolution on Binomial distribution sample (gemm error)

[Jesse Livezey]

FYI, I created an issue to improve the error messages
https://github.com/Theano/Theano/issues/6167

On Tuesday, July 18, 2017 at 12:18:53 PM UTC-7, Jesse Livezey wrote:
>
> The conv2d operation doesn't support int64 (th_sampled) and it looks like 
> it doesn't fail gracefully with a sensible error message when the GpuCorrMM 
> op is used.
>
> If you cast th_sampled to float32 it should work fine. You'll also need to 
> cast kernel.
>
> On Tuesday, July 18, 2017 at 2:07:36 AM UTC-7, David Anderson wrote:
>>
>> Hi there!
>>
>> I'm implementing a convolutional operation and I'm getting an unexpected 
>> error when I try perform a convolution on a Binomial sampled tensor.
>>
>> The error is: 
>> RuntimeError: GpuCorrMM forward encountered an error running gemm: 5
>>
>> The error can be re-created with the following code (At least on my 
>> machine it can):
>>
>> import numpy as np
>> import theano as th
>> from theano import tensor as T
>> from theano.tensor.shared_randomstreams import RandomStreams
>>
>> rng = np.random.RandomState()
>> theano_rng = RandomStreams(rng.randint(2 ** 30))
>>
>> th_input = T.tensor4()
>> th_filter = T.tensor4()
>>
>> th_sampled = theano_rng.binomial(size=th_input.shape, n=1, p=th_input)
>> th_output = T.nnet.conv2d(th_sampled, th_filter)
>>
>> op = th.function(
>> inputs=[th_input, th_filter],
>> outputs=th_output
>> )
>>
>> input_sample = np.random.rand(1, 1, 28, 28)
>> kernel = np.random.rand(1, 1, 6, 6)
>>
>> op(input_sample, kernel)
>>
>>
>> Interestingly, the error is NOT shown for other distribution samples, 
>> like theano_rng.normal(), which has type RandomFunction{normal}.1 
>> instead of RandomFunction{binomial}.1
>>
>> For what its worth, my THEANO_FLAGS are as follows:
>> floatX=float64,device=cuda,nvcc.flags=-D_FORCE_INLINES,
>> exception_verbosity=high
>>
>> The rest of the stack trace is as follows:
>> Traceback (most recent call last):
>>   File "tmp2.py", line 23, in 
>> op(input_sample, kernel)
>>   File 
>> "/home/dave/miniconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
>>  
>> line 898, in __call__
>> storage_map=getattr(self.fn, 'storage_map', None))
>>   File 
>> "/home/dave/miniconda2/lib/python2.7/site-packages/theano/gof/link.py", 
>> line 325, in raise_with_op
>> reraise(exc_type, exc_value, exc_trace)
>>   File 
>> "/home/dave/miniconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
>>  
>> line 884, in __call__
>> self.fn() if output_subset is None else\
>> RuntimeError: GpuCorrMM forward encountered an error running gemm: 5
>> Apply node that caused the error: GpuCorrMM{valid, (1, 1), (1, 
>> 1)}(GpuContiguous.0, GpuContiguous.0)
>> Toposort index: 11
>> Inputs types: [GpuArrayType(int64, (False, False, False, False)), 
>> GpuArrayType(float64, (False, False, False, False))]
>> Inputs shapes: [(1, 1, 28, 28), (1, 1, 6, 6)]
>> Inputs strides: [(6272, 6272, 224, 8), (288, 288, 48, 8)]
>> Inputs values: ['not shown', 'not shown']
>> Inputs type_num: [7, 12]
>> Outputs clients: [[HostFromGpu(gpuarray)(GpuCorrMM{valid, (1, 1), (1, 
>> 1)}.0)]]
>>
>> Debugprint of the apply node: 
>> GpuCorrMM{valid, (1, 1), (1, 1)} [id A] <GpuArrayType(int64, 
>> (False, False, False, False))> ''   
>>  |GpuContiguous [id B] <GpuArrayType(int64, (False, False, False, 
>> False))> ''   
>>  | |GpuFromHost [id C] <GpuArrayType(int64, (False, False, 
>> False, False))> ''   
>>  |   |RandomFunction{binomial}.1 [id D] <TensorType(int64, 4D)> ''   
>>  | | [id E] 
>>  | |MakeVector{dtype='int64'} [id F] <TensorType(int64, vector)> ''   
>>  | | |Shape_i{0} [id G] <TensorType(int64, scalar)> ''   
>>  | | | |<TensorType(float64, 4D)> [id H] <TensorType(float64, 4D)>
>>  | | |Shape_i{1} [id I] <TensorType(int64, scalar)> ''   
>>  | | | |<TensorType(float64, 4D)> [id H] <TensorType(float64, 4D)>
>>  | | |Shape_i{2} [id J] <TensorType(int64, scalar)> ''   
>>  | | | |<TensorType(float64, 4D)> [id H] <TensorType(float64, 4D)>
>>  | | |Shape_i{3} [id K] <TensorType(int64, scalar)> ''   
>>  | |   |<TensorType(float64, 4D)> [id H] <TensorType(float64, 4D)>
>>  | |TensorConstant{1} [id L] <TensorType(int8, scalar)>
>>  | |<Tens

[theano-users] Re: Error performing 2D convolution on Binomial distribution sample (gemm error)

[Jesse Livezey]

The conv2d operation doesn't support int64 (th_sampled) and it looks like 
it doesn't fail gracefully with a sensible error message when the GpuCorrMM 
op is used.

If you cast th_sampled to float32 it should work fine. You'll also need to 
cast kernel.

On Tuesday, July 18, 2017 at 2:07:36 AM UTC-7, David Anderson wrote:
>
> Hi there!
>
> I'm implementing a convolutional operation and I'm getting an unexpected 
> error when I try perform a convolution on a Binomial sampled tensor.
>
> The error is: 
> RuntimeError: GpuCorrMM forward encountered an error running gemm: 5
>
> The error can be re-created with the following code (At least on my 
> machine it can):
>
> import numpy as np
> import theano as th
> from theano import tensor as T
> from theano.tensor.shared_randomstreams import RandomStreams
>
> rng = np.random.RandomState()
> theano_rng = RandomStreams(rng.randint(2 ** 30))
>
> th_input = T.tensor4()
> th_filter = T.tensor4()
>
> th_sampled = theano_rng.binomial(size=th_input.shape, n=1, p=th_input)
> th_output = T.nnet.conv2d(th_sampled, th_filter)
>
> op = th.function(
> inputs=[th_input, th_filter],
> outputs=th_output
> )
>
> input_sample = np.random.rand(1, 1, 28, 28)
> kernel = np.random.rand(1, 1, 6, 6)
>
> op(input_sample, kernel)
>
>
> Interestingly, the error is NOT shown for other distribution samples, like 
> theano_rng.normal(), 
> which has type RandomFunction{normal}.1 instead 
> of RandomFunction{binomial}.1
>
> For what its worth, my THEANO_FLAGS are as follows:
> floatX=float64,device=cuda,nvcc.flags=-D_FORCE_INLINES,exception_verbosity
> =high
>
> The rest of the stack trace is as follows:
> Traceback (most recent call last):
>   File "tmp2.py", line 23, in 
> op(input_sample, kernel)
>   File 
> "/home/dave/miniconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
>  
> line 898, in __call__
> storage_map=getattr(self.fn, 'storage_map', None))
>   File 
> "/home/dave/miniconda2/lib/python2.7/site-packages/theano/gof/link.py", 
> line 325, in raise_with_op
> reraise(exc_type, exc_value, exc_trace)
>   File 
> "/home/dave/miniconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
>  
> line 884, in __call__
> self.fn() if output_subset is None else\
> RuntimeError: GpuCorrMM forward encountered an error running gemm: 5
> Apply node that caused the error: GpuCorrMM{valid, (1, 1), (1, 
> 1)}(GpuContiguous.0, GpuContiguous.0)
> Toposort index: 11
> Inputs types: [GpuArrayType(int64, (False, False, False, False)), 
> GpuArrayType(float64, (False, False, False, False))]
> Inputs shapes: [(1, 1, 28, 28), (1, 1, 6, 6)]
> Inputs strides: [(6272, 6272, 224, 8), (288, 288, 48, 8)]
> Inputs values: ['not shown', 'not shown']
> Inputs type_num: [7, 12]
> Outputs clients: [[HostFromGpu(gpuarray)(GpuCorrMM{valid, (1, 1), (1, 
> 1)}.0)]]
>
> Debugprint of the apply node: 
> GpuCorrMM{valid, (1, 1), (1, 1)} [id A]  False, False, False))> ''   
>  |GpuContiguous [id B]  False))> ''   
>  | |GpuFromHost [id C]  False, False))> ''   
>  |   |RandomFunction{binomial}.1 [id D]  ''   
>  | | [id E] 
>  | |MakeVector{dtype='int64'} [id F]  ''   
>  | | |Shape_i{0} [id G]  ''   
>  | | | | [id H] 
>  | | |Shape_i{1} [id I]  ''   
>  | | | | [id H] 
>  | | |Shape_i{2} [id J]  ''   
>  | | | | [id H] 
>  | | |Shape_i{3} [id K]  ''   
>  | |   | [id H] 
>  | |TensorConstant{1} [id L] 
>  | | [id H] 
>  |GpuContiguous [id M]  False))> ''   
>|GpuFromHost [id N]  False, False))> ''   
>  |Subtensor{::, ::, ::int64, ::int64} [id O]  
> ''   
>| [id P] 
>|Constant{-1} [id Q] 
>|Constant{-1} [id Q] 
>
> Storage map footprint:
>  - GpuContiguous.0, Shape: (1, 1, 28, 28), ElemSize: 8 Byte(s), TotalSize: 
> 6272 Byte(s)
>  - , Input, Shape: (1, 1, 28, 28), ElemSize: 8 
> Byte(s), TotalSize: 6272 Byte(s)
>  - GpuContiguous.0, Shape: (1, 1, 6, 6), ElemSize: 8 Byte(s), TotalSize: 
> 288 Byte(s)
>  - , Input, Shape: (1, 1, 6, 6), ElemSize: 8 
> Byte(s), TotalSize: 288 Byte(s)
>  - Constant{-1}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
>  - TensorConstant{1}, Shape: (), ElemSize: 1 Byte(s), TotalSize: 1.0 
> Byte(s)
>  

[theano-users] Re: About theano.function inside for loop

[Jesse Livezey]

Yes, you should be able to just call theano.function(...) before the loops.

On Wednesday, July 12, 2017 at 4:13:33 AM UTC-7, Kelvin Chiu wrote:
>
> for x in range(x_range):
> for y in range(y_range):
> t_test_set_x = theano_translation(test_set_x, x, y, borrow=True)
> predict_model = theano.function(inputs=[index],
> outputs=layer3.errors(y),
> givens={layer0.input: 
> t_test_set_x[index * 500: (index + 1) * 500],
> y: test_set_y[index * 500: 
> (index + 1) * 500]})
> for batch_value in range(0, 20, 1):
> temp_predicted_values = predict_model(batch_value)
> predicted_values = temp_predicted_values + predicted_values
>
>
> This is part of my source code. Now, the theano function is put inside 2 for 
> loops. And my test set is updated in every loop. Is there anyway to put the
>  theano function outside the for loop so that i can speed up the 
> computational process ? 
>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: How to implement binary activation in theano?

[Jesse Livezey]

Do you need to take derivatives through the activation? If not, then you 
could use switch, i.e.

x = some theano variable
threshold = .5
x_binary = T.switch(x > theshold, 1., 0.)

On Wednesday, July 12, 2017 at 10:27:32 AM UTC-7, zxzh...@gmail.com wrote:
>
> In the binarized network github code (), Matthieu used stochastic 
> binarization. I'm wondering how to define just a simple binary activation 
> instead of stochastic in theano?
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Re: How create activation function from scratch in python

[Jesse Livezey]

Here you're treating val like it is a symbolic theano variable
T.log(val)

But here you're treating it like a numpy array and passing it into a 
compiled theano function
return f_switch(val, 0, val, val)

Maybe you're intending to just return the function f_switch and then call 
it with values?


On Wednesday, July 5, 2017 at 3:02:58 PM UTC-7, nouiz wrote:
>
> Give the full error message. Without our I can't help.
>
> Fred
>
> Le mer. 5 juil. 2017 12:33, Bruno Messias  > a écrit :
>
>> I'  need call "custom" function with a given variable  x, such that
>>
>> type(x)
>>
>>
>> On Wednesday, July 5, 2017 at 12:53:22 PM UTC-3, Bruno Messias wrote:
>>>
>>> For didactic reasons, I am trying to implement a  "activation"  function
>>>
>>>
>>> a, x, y = T.matrices("a", 'x','y')
>>> b = T.scalars("b")
>>> def  custom(val):
>>>
>>> T.log(val)
>>> 
>>> 
>>> z_switch = T.switch(T.gt(a,b), T.true_div(T.add(T.pow(x, qEff),0), 
>>> 2), T.log(y))
>>>
>>> f_switch = theano.function([a, b, x, y], z_switch,
>>>mode=theano.Mode(linker='vm'))
>>> return f_switch(val, 0, val, val)
>>>
>>> Then I get the following error
>>>
>>> Expected an array-like object, but found a Variable: maybe you are trying 
>>> to call a function on a (possibly shared) variable instead of a numeric 
>>> array?
>>>
>>> Repeating> this is only for didactic purposes. There are any good tutorial 
>>> about this?
>>>
>>> -- 
>>
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "theano-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to theano-users...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: How can implemeant BackPropagation through time in Theano?

[Jesse Livezey]

You can use the scan function to create RNN architectures.

http://deeplearning.net/software/theano/library/scan.html

On Sunday, June 18, 2017 at 4:13:44 PM UTC-7, Sunjeet Jena wrote:
>
> I am building a multi-layer RNN network and thus need a way to back 
> propagate through time in Theano. Does theano automatically knows how to 
> unfold the network as a feed forward network?
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Modify theano MLP class object with multiple outputs

[Jesse Livezey]

Theano functions can have multiple outputs. What error are you getting?

On Monday, May 29, 2017 at 2:52:17 PM UTC-7, rogelio andrade wrote:
>
> Hi
>
>
> I am using this theano code MLP 
>  which implements a 
> variational AE. I am interested on getting self.mu and self.var when the 
> MLP is used in the decoder. However, I am not sure what is the easiest way 
> to do such modification in this code (and on the vae.py), basically I need 
> something like this:
>
>
> # for use as decoder
> if y:
> assert(eps is None)
> # XXX specific to [0, 1] outputs
> self.out = [T.nnet.sigmoid(self.mu),self.mu,self.var]
> self.cost = -T.sum(log_diag_mvn(self.out, self.var)(y))
>
>
> but of course this does not work. Is it a way to get multiple outputs as 
> above? or how can I get the mu and var in the decoder step?
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: tensor grad with respect to shared variable

[Jesse Livezey]

t will have dtype "int64" because 5 is an integer. If you do
t = theano.shared(5.,'t') # added period after 5
it should work

On Thursday, May 18, 2017 at 2:59:17 AM UTC-7, phamminhquang pham wrote:
>
>  I have tried following code and it's result confuse me.
>
> import theano
> import theano.tensor as T
> 
> import theano
> import theano.tensor as T
>
> t = theano.shared(5,'t')
> y = t ** 2
> grad = T.grad(y, wrt = t)
> f = theano.function([],y)
> g = theano.function([],grad)
> print(f())
> print(g())
>
> And i got 25 and 0.0 in return. However, the gradient must be 10. I think 
> there's some detail in tensor grad that i dont know. Could you please 
> explain to me what happened?
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Classic error: Expected an array-like object, but found a Variable

[Jesse Livezey]

That isn't a normal way to use theano. Have you read about creating theano 
functions with "updates" dictionaries?

There are simple examples here:
http://deeplearning.net/software/theano/tutorial/examples.html#using-shared-variables

On Sunday, May 14, 2017 at 3:05:01 AM UTC-7, Mohamed Akrout wrote:
>
> Hi,
>
> I am new to Theano. My problematic code is the following:
>
> z0 = float32(random.randn(1, 1))
> z = shared(z0)
>
> x0 = float32(random.randn(N, 1))
> x = shared(x0)
>
> wo = shared(zeros((N, 1), dtype=float32))
>
> z.set_value(T.dot(wo.T , x)) # here is the problem
>
>
> This gives me the error: Expected an array-like object, but found a 
> Variable.
>
> I understand that z is a 1x1 numpy array and T.dot(wo.T , x) is a 1x1 
> vector but I did not succeed to find a way to assign the 1x1 vector to z.
>
> Do you have any idea to solve this problem?
>
> Thank you for your help!
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Is there a way to speed this operation in theano

[Jesse Livezey]

I see, you can use batched_dot for that. I wrote a gist which compares the 
numpy matmul, theano batch_dot, and theano multiply and sum approaches.
https://gist.github.com/JesseLivezey/42cabcf87aa0033410f7520933942127

On GPU, the multiply and sum seems to be fastest, but it will also use more 
memory.


On Monday, May 8, 2017 at 1:30:33 AM UTC-7, Šarūnas S. wrote:
>
> Currently, I have 3 approaches that are portable to theano:
>
> # 3D example
> axis = 0
> prob = np.random.random( ( 1, 1000, 50 ) )
> cases = np.random.random( ( 1000, 1000, 50 ) )
>
> # Elementwise + sum
> for i in xrange( 100 ):
> result = ( cases * prob ).sum( axis=1-axis, keepdims=True )
>
> # Loop version
> result = np.zeros( ( 1000, 1, 50 ) )
> for i in xrange( 5 ):
> result[ :, :, i ] = np.dot( prob[ :, :, i ], cases[ :, :, i ] )
>
> # Block diagonal sparse dot version
> prob_big = np.zeros( ( 1, 1000, 50, 50 ) )
> cases_big = np.zeros( ( 1000, 1000, 50, 50 ) )
>
> for i in xrange( 50 ):
> prob_big[ :, :, i, i ] = prob[ :, :, i, i ]
> cases_big[ :, :, i, i ] = prob[ :, :, i, i ]
>
> intermediate = np.tensordot( prob_big, cases_big, axes=[ [ 0 ], [ 1 ] ] )
> result = np.zeros( 1000, 1, 50 )
> for i in range( 50 ):
> result[ :, :, i ] = intermediate[ :, :, i, i ]
>
> I think the the one which would structure this as a sparse block diagonal 
> matrix would work best since I've seen some support for the block sparse 
> matrices. However, it looks like I would still need some loop for 
> blocksparse to iterate over all the blocks. Is there a way to somehow do 
> all the blocks at once and collect the diagonal without using scan? 
>
> On Saturday, 6 May 2017 10:41:06 UTC+3, Šarūnas S. wrote:
>>
>> I have tried that, but to no avail. The problem is that I have to 
>> multiply on 2 axes, but sum only on 1. 
>>
>> On Friday, 5 May 2017 19:23:12 UTC+3, Jesse Livezey wrote:
>>>
>>> I think tensordot should do what you want
>>>
>>> http://deeplearning.net/software/theano/library/tensor/basic.html#theano.tensor.tensordot
>>> something like
>>> result = T.tensordot(prob, cases, axes=1)
>>>
>>>
>>>
>>> On Friday, May 5, 2017 at 3:17:14 AM UTC-7, Šarūnas S. wrote:
>>>>
>>>> I was shown that in *numpy* I could speed it up in the following way:
>>>>
>>>> result = np.einsum('ijk,ijk->ik', prob, cases)[:,None,:]
>>>>
>>>>
>>>> result = np.matmul(prob.transpose(2,0,1), cases.T).T
>>>>
>>>>
>>>> Bot give me the expected speedup in *numpy*, but neither is 
>>>> implemented in *Theano*. Is there a way to do the same in *Theano* on 
>>>> the *GPU*?
>>>>
>>>>
>>>>
>>>> On Friday, 5 May 2017 11:15:26 UTC+3, Šarūnas S. wrote:
>>>>>
>>>>> In my current theano script the bottleneck is equivalent to the 
>>>>> following numpy code:
>>>>>
>>>>> import numpy as np
>>>>>
>>>>> # 3D example
>>>>> axis = 0
>>>>> prob = np.random.random( ( 1, 1000, 50 ) )
>>>>> cases = np.random.random( ( 1000, 1000, 50 ) )
>>>>>
>>>>> start = time.time(  )
>>>>> for i in xrange( 1000 ):
>>>>> result = ( cases * prob ).sum( axis=1-axis, keepdims=True )
>>>>> print '3D naive method took {} seconds'.format( time.time() - start )
>>>>> print result.shape
>>>>> print
>>>>>
>>>>> I had seen in 2D case that replacing elementwise+sum with a dot 
>>>>> product gave me 5x speedup. Are there any theano matrix operations that 
>>>>> could help me out here? 
>>>>>
>>>>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Is there a way to speed this operation in theano

[Jesse Livezey]

I think tensordot should do what you want
http://deeplearning.net/software/theano/library/tensor/basic.html#theano.tensor.tensordot
something like
result = T.tensordot(prob, cases, axes=1)



On Friday, May 5, 2017 at 3:17:14 AM UTC-7, Šarūnas S. wrote:
>
> I was shown that in *numpy* I could speed it up in the following way:
>
> result = np.einsum('ijk,ijk->ik', prob, cases)[:,None,:]
>
>
> result = np.matmul(prob.transpose(2,0,1), cases.T).T
>
>
> Bot give me the expected speedup in *numpy*, but neither is implemented 
> in *Theano*. Is there a way to do the same in *Theano* on the *GPU*?
>
>
>
> On Friday, 5 May 2017 11:15:26 UTC+3, Šarūnas S. wrote:
>>
>> In my current theano script the bottleneck is equivalent to the following 
>> numpy code:
>>
>> import numpy as np
>>
>> # 3D example
>> axis = 0
>> prob = np.random.random( ( 1, 1000, 50 ) )
>> cases = np.random.random( ( 1000, 1000, 50 ) )
>>
>> start = time.time(  )
>> for i in xrange( 1000 ):
>> result = ( cases * prob ).sum( axis=1-axis, keepdims=True )
>> print '3D naive method took {} seconds'.format( time.time() - start )
>> print result.shape
>> print
>>
>> I had seen in 2D case that replacing elementwise+sum with a dot product 
>> gave me 5x speedup. Are there any theano matrix operations that could help 
>> me out here? 
>>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] improving fft performance on cpu

[Jesse Livezey]

>From tests I've done, the MKL fft library is comparable and sometimes 
faster than the FFTW package. Both are much faster than the numpy fft. It's 
available in the accelerate package from Continuum (mkl conda package only 
has blas).

It also looks like intel has a free conda channel, which builds scipy and 
numpy against MKL
https://software.intel.com/en-us/forums/intel-distribution-for-python/topic/713736

On Wednesday, April 19, 2017 at 10:54:33 AM UTC-7, nouiz wrote:
>
> Blas won't help for fft. So you would need a faster fft library then what 
> we use and modify the perform() method of those ops to use that new lib.
>
> FFTW is one possible faster implementation. There is others. I can't 
> comment on which one would be better. Search the web, I recall seeing some 
> people comparing fft lib that is available in python.
>
> If you modify those ops for that, pushing upstream those change would be 
> great.
>
> Fred
>
>
>
> On Tue, Apr 18, 2017 at 12:16 PM  wrote:
>
>> Hi,
>>
>> I have implemented a layer which uses functions theano.tensor.fft.rfft 
>> and theano.tensor.fft.irfft. What might be the best way to improve the 
>> speed of that layer on cpu? Installing FFTW, an optimized BLAS library?
>>
>> Thanks
>> Cha.
>>
>> -- 
>>
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "theano-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to theano-users...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: RuntimeError: Mixed dnn version. The header is version 5105 while the library is version 5110.

[Jesse Livezey]

Sounds like the cudnn header and libraries are not consistent. When you 
install cudnn, did you move all of the files into the correct cuda folders?

On Monday, April 17, 2017 at 8:30:03 PM UTC-7, Robert Lee wrote:
>
> I'm trying to get theano to work with keras.  My program runs fine with 
> tensorflow but when I switch to theano I get the above error message.  My 
> theano version is 0.9.0.  I'd appreciate any help in figuring this out.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Decrypting theano scan error message

[Jesse Livezey]

It looks like originally x was a list of ints. Your x is a vector. I think 
what you currently have written will take one element of the vector x per 
time step and so T.dot(W_xh, x_t) will be a matrix times a scalar which is 
probably leading to your shape problems. Do you have a sequence of vectors 
for x or just one? If you have a sequence, x should come from T.matrix and 
if you just have one, you should just pre-compute the dot product and pass 
it as a non-sequence.

If x is a matrix, scan will iterate over the rows of the matrix.


On Tuesday, March 28, 2017 at 10:58:04 AM UTC-7, mrinmoy maity wrote:
>
> I'm trying to verify the vanilla RNN model using Theano but is seeing the 
> error I am not being able to understand. Any help will be much appreciated.
>
>  
> Following is the code snippet for RNN:
> -
> W_xh, W_hy, W_hh, b_h, b_y = self.params
> x = T.vector('x')
> y = T.vector('y')
>def forward_prop_step(x_t, h_t_prev, W_xh, W_hy, W_hh, b_h, b_y):
> h_t = T.tanh(W_xh[:, x_t] + T.dot(W_hh, h_t_prev) + b_h) 
> ...(1)
> #h_t = T.tanh(T.dot(W_xh, x_t) + T.dot(W_hh, h_t_prev) + 
> b_h).(2)
> o_t = T.nnet.softmax(T.dot(W_hy, h_t) + b_y)
> return [o_t[0], h_t]
> h_0 = T.zeros(self.n_hidden)
> [o,h], _ = theano.scan(
> forward_prop_step,
> sequences=x,
> outputs_info=[None, h_0],
> non_sequences=[W_xh, W_hy, W_hh, b_h, b_y],
> truncate_gradient=self.bptt_truncate,
> strict=True)
> ---
>
> In line (1) and (2), the only difference is replacing W_xh[:, x_t] with 
> actual dot product T.dot(W_xh, x_t). When x_t is one hot encoding vector, 
> column selection specified in (1) works fine. But if input is a vector of 
> floating point numbers, we need to use what specified in (2). However, 
> using (2) throws the following error:
>
> ValueError: When compiling the inner function of scan (the function called 
> by scan in each of its iterations) the following error has been 
> encountered: The initial state (`outputs_info` in scan nomenclature) of 
> variable IncSubtensor{Set;:int64:}.0 (argument number 1) has 2 
> dimension(s), while the corresponding variable in the result of the inner 
> function of scan (`fn`) has 2 dimension(s) (it should be one less than the 
> initial state). For example, if the inner function of scan returns a vector 
> of size d and scan uses the values of the previous time-step, then the 
> initial state in scan should be a matrix of shape (1, d). The first 
> dimension of this matrix corresponds to the number of previous time-steps 
> that scan uses in each of its iterations. In order to solve this issue if 
> the two varialbe currently have the same dimensionality, you can increase 
> the dimensionality of the variable in the initial state of scan by using 
> dimshuffle or shape_padleft. 
>
>
> Seems like there's some problem with initialization 'h_0'.
>
> My questions are:
> 1. Is there any workaround to avoid the issue like how 'h_0' should be 
> initialized?
> 2. Why does it work for one hot encoding vector in line (1) but not in 
> line (2)?
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Re: IfElse GPU version

[Jesse Livezey]

>
> I have decided to precompile a general graph in which all the possible 
> graphs are nested. Then during realtime I would set which parts of the 
> general graph to use using the *allowed_branch* variables and *if* nodes. 
> Since afaik ifs are evaluated lazily in each case I would only be using the 
> relevant part of the graph so my computational cost is minimal.


Have you considered precompiling all possible graphs individually and then 
just using python conditionals to choose a graph? Maybe this won't work for 
your system, but it might be easier to get right.

On Saturday, March 25, 2017 at 2:21:27 AM UTC-7, Šarūnas S. wrote:
>
> Nouiz sorry I understand what you were refering by is constant. I've 
> mislead you with my example. 
>
> This is a more realistic example:
>
> import theano as th
> import theano.tensor as T
>
> allowed_branch = th.shared( np.cast['float32']( 0 ) )
>
> x = T.matrix('x')
> y = T.matrix('y')
> f = x ** 2 + y ** 2 + 2 * x * y  
>  
> result = th.ifelse.ifelse( T.gt( allowed_branch, T.constant( 0 ) ), f, 
> T.zeros( (2,2) ) )
> 
>
>
> I am working on a realtime system which in a given situation will 
> constructs a relevant computational graph, compute its result and display 
> it. 
> However, the graphs are relatively big and each of their compilation takes 
> too long so I cant compile realtime. Thus I have to somehow precompile. 
>
> I have decided to precompile a general graph in which all the possible 
> graphs are nested. Then during realtime I would set which parts of the 
> general graph to use using the *allowed_branch* variables and *if* nodes. 
> Since afaik ifs are evaluated lazily in each case I would only be using the 
> relevant part of the graph so my computational cost is minimal.
>
>
> On Saturday, 25 March 2017 10:04:21 UTC+1, Šarūnas S. wrote:
>>
>> I suspect that ifelse is running on GPU because this is the profile I get
>>
>> ==
>>   Message: Sum of all(44) printed profiles at exit excluding Scan op 
>> profile.
>>   Time in 95 calls to Function.__call__: 2.309995e-01s
>>   Time in Function.fn.__call__: 2.25e-01s (99.567%)
>>   Time in thunks: 2.307765e-01s (99.903%)
>>   Total compile time: 1.360100e+01s
>> Number of Apply nodes: 416
>> Theano Optimizer time: 6.314001e+00s
>>Theano validate time: 9.200015e-01s
>> Theano Linker time (includes C, CUDA code generation/compiling): 
>> 1.169000e+00s
>>Import time 2.799892e-02s
>>Node make_thunk time 1.108999e+00s
>>Node GpuElemwise{Composite{(i0 * ((i1 * i2) + (i1 * 
>> i3)))}}[(0, 2)](CudaNdarrayConstant{0.5}, 
>> CudaNdarrayConstant{0.83313465}, GpuCAReduce{add}{1,1}.0, 
>> GpuCAReduce{add}{1,1}.0) time 6.69e-03s
>>Node GpuElemwise{Composite{(-minimum(i0, maximum(minimum(i0, 
>> (maximum((i1 - i2), i3) + i2)), (((i1 + i2) * i4) + 
>> i1},no_inplace}(, 
>> , , 
>> CudaNdarrayConstant{120.0}, ) time 
>> 4.999876e-03s
>>Node GpuElemwise{mul,no_inplace}(> matrix)>, GpuElemwise{TrueDiv}[(0, 0)].0) time 4.000187e-03s
>>Node HostFromGpu() time 
>> 3.49e-03s
>>Node GpuElemwise{Mul}[(0, 1)](GpuDimShuffle{x,x}.0, 
>> GpuDimShuffle{x,0}.0) time 3.49e-03s
>>
>> Time in all call to theano.grad() 0.00e+00s
>> Time since theano import 28.959s
>> Class
>> ---
>> <% time> <#call> <#apply> 
>> 
>>   55.4%55.4%   0.128s   8.71e-05s C 1468 301   
>> theano.sandbox.cuda.basic_ops.GpuElemwise
>>   25.6%81.0%   0.059s   1.03e-04s C  571 106   
>> theano.sandbox.cuda.basic_ops.GpuCAReduce
>>9.1%90.1%   0.021s   3.72e-05s C  564 150   
>> theano.sandbox.cuda.basic_ops.HostFromGpu
>>5.6%95.7%   0.013s   6.04e-06s Py2148 168   
>> theano.ifelse.IfElse
>>3.5%99.1%   0.008s   2.16e-04s C   37   4   
>> theano.compile.ops.DeepCopyOp
>>0.4%99.6%   0.001s   1.60e-06s C  623 122   
>> theano.sandbox.cuda.basic_ops.GpuDimShuffle
>>0.4%   100.0%   0.001s   1.97e-06s C  506 110   
>> theano.tensor.elemwise.Elemwise
>>... (remaining 0 Classes account for   0.00%(0.00s) of the runtime)
>>
>> Ops
>> ---
>> <% time> <#call> <#apply> > name>
>>   16.9%16.9%   0.039s   1.22e-04s C  319   58   
>> GpuElemwise{mul,no_inplace}
>>   10.0%26.9%   0.023s   1.49e-04s C  155   30   
>> GpuCAReduce{add}{1,0}
>>9.1%36.0%   0.021s   3.72e-05s C  564  150   
>> HostFromGpu
>>8.2%44.2%   0.019s   1.23e-04s C  154   30   
>> GpuCAReduce{add}{0,1}
>>6.9%51.1%   0.016s   6.61e-05s C  

Re: [theano-users] What i do wrong when I use theano in production?

[Jesse Livezey]

Rather than using givens, and getting the batch through indexing, you can 
just have a theano variable be an input to your function. You should only 
need to compile the function once if your model does not change. You should 
be able to do something like this (you may need to modify this since I 
don't know what the rest of the code looks like)


predict_by_model = theano.function(inputs=[self.x],
   outputs=self.classifier.predict(self.y))

sentences = ['first', 'second', 'third']

for i in sentences:
new_data_by_i = get_data(i) # this is new data

res = predict_my_model(new_data_by_i)

where new_data_by_i should be a matrix or vector of data, not an index.

On Saturday, March 25, 2017 at 12:33:38 AM UTC-7, Нурислам Алимов wrote:
>
> Thanks Fred. 
>
> So, I am not fully understand how can I pickle compiled function. Because 
> lokk please in my code:
>
> def predict_by_model(data):
> test_model_predict = theano.function(inputs=[self.start_idx, self.
> end_idx],
>  outputs=self.classifier.predict(self.y),
>  givens={
>  self.x: data_x[self.start_idx:self.end_idx],
>  self.y: data_y[self.start_idx:self.end_idx]})
>
> test_predictions = test_model_predict(0, len(test_set[1]))
>
> sentences = ['first', 'second', 'third']
>
> for i in sentences:
> new_data_by_i = get_data(i) # this is new data
>
> res = predict_my_model(new_data_by_i)
>
> Everytime in the loop I have new data. I think I can't precompile 
> functions because I don't know what data I will get in next time? 
>
> Or I'm wrong and there is another way.
>
> Thanks for you help.
>
>
>
> суббота, 25 марта 2017 г., 1:09:14 UTC+3 пользователь nouiz написал:
>>
>> It depend of your production environment. But you can start a server with 
>> multiple process and they compile the function when the process is started, 
>> but you reuse it for many function call.
>>
>> You can also pickle the Theano function and have the process unpickle it. 
>> It would be faster then recompile it. This do not remove any tho 
>> dependencies.
>>
>> Fred
>>
>> Le ven. 24 mars 2017 05:47, Нурислам Алимов  a 
>> écrit :
>>
>>> I use this code in parallel. Is it correct?
>>>
>>>
>>> test_model_predict = theano.function(inputs=[self.start_idx, self.
>>> end_idx],
>>>  outputs=self.classifier.predict(self.y),
>>>  givens={
>>>  self.x: test_set_x[self.start_idx:self.end_idx],
>>>  self.y: test_set_y[self.start_idx:self.end_idx]})
>>>
>>>
>>> test_predictions = test_model_predict(0, len(test_set[1]))
>>>
>>> How can I call predictions of model without use theano.function - 
>>> because it is not thread safe function as I know and it is very long.
>>>
>>> Please help.
>>>
>>> -- 
>>>
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "theano-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to theano-users...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: about conv2d on CPU

[Jesse Livezey]

That is correct as of theano 0.8 (I think).

If you use the bleeding edge version of theano, you can let CorrMM use 
openmp to parallelize across batches. If you have more than 2 cores, this 
should give additional speedup. GPUs are going to be much faster than CPUs 
generally, if you have large batches and lots of cores, CPUs can catch up a 
bit, but GPUs are still going to be faster.

On Monday, March 20, 2017 at 11:59:52 PM UTC-7, C. Ng wrote:
>
> Hi,
>
> Just want to confirm that theano.tensor.nnet.conv2d uses CorrMM (not the 
> legacy convolution) by default in CPU mode ?
>
> I was hoping that forward prop (doing inference only, no training) using 
> CPU for convolution might be as fast as GPU (using CorrMM), given my batch 
> size is only 10. But using GPU is still quite a bit faster.  
>
>
>
>
>  
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Some help optimizing a function involving 1D dot products for multidimensional tensors

[Jesse Livezey]

If I'm understanding your code correctly, you should be able to use 
tensordot
http://deeplearning.net/software/theano/library/tensor/basic.html#theano.tensor.tensordot
rather than doing the multiply and sum.

On Thursday, March 16, 2017 at 10:59:14 AM UTC-4, Eelke Spaak wrote:
>
> Apologies for the messed up profiling code, here is attempt 2:
>
> Class
> ---
> <% time> <#call> <#apply> 
> 
>   46.2%46.2%  10.971s   2.74e-05s C   400764  42   
> theano.sandbox.cuda.basic_ops.GpuElemwise
>   29.9%76.0%   7.098s   3.72e-05s C   190840  20   
> theano.sandbox.cuda.basic_ops.GpuCAReduce
>7.2%83.2%   1.699s   1.48e-05s C   114504  12   
> theano.sandbox.cuda.blas.GpuDot22
>3.8%87.0%   0.911s   4.78e-05s C19084   2   
> theano.sandbox.cuda.basic_ops.GpuJoin
>3.8%90.9%   0.907s   5.59e-06s C   162214  17   
> theano.sandbox.cuda.basic_ops.GpuFromHost
>2.9%93.8%   0.700s   1.05e-05s C66794   7   
> theano.sandbox.cuda.basic_ops.HostFromGpu
>2.1%95.9%   0.501s   1.14e-06s C   438932  46   
> theano.sandbox.cuda.basic_ops.GpuReshape
>1.5%97.4%   0.348s   1.46e-06s C   238550  25   
> theano.tensor.elemwise.Elemwise
>1.4%98.7%   0.327s   3.43e-05s C 9542   1   
> theano.sandbox.cuda.blas.GpuGemv
>0.4%99.2%   0.097s   9.28e-07s C   104962  11   
> theano.sandbox.cuda.basic_ops.GpuDimShuffle
>0.3%99.5%   0.081s   1.06e-06s C76336   8   
> theano.sandbox.cuda.basic_ops.GpuSubtensor
>0.2%99.7%   0.042s   4.35e-06s C 9542   1   
> theano.tensor.basic.Join
>0.1%99.8%   0.033s   8.62e-07s C38168   4   
> theano.tensor.elemwise.DimShuffle
>0.1%99.9%   0.019s   9.75e-07s C19084   2   
> theano.tensor.subtensor.Subtensor
>0.1%99.9%   0.015s   1.54e-06s C 9542   1   
> theano.sandbox.cuda.basic_ops.GpuAllocEmpty
>0.1%   100.0%   0.012s   6.46e-07s C19084   2   
> theano.compile.ops.ViewOp
>... (remaining 0 Classes account for   0.00%(0.00s) of the runtime)
>
> Ops
> ---
> <% time> <#call> <#apply>  name>
>   24.7%24.7%   5.860s   6.14e-05s C 95420   10   
> GpuElemwise{mul,no_inplace}
>   17.6%42.2%   4.173s   1.09e-04s C 381684   
> GpuCAReduce{add}{1,1,1}
>7.2%49.4%   1.699s   1.48e-05s C 114504   12   
> GpuDot22
>4.1%53.5%   0.974s   2.55e-05s C 381684   
> GpuCAReduce{add}{0,1,0}
>4.1%57.6%   0.972s   2.55e-05s C 381684   
> GpuCAReduce{add}{0,1}
>3.8%61.4%   0.911s   4.78e-05s C 190842   
> GpuJoin
>3.8%65.2%   0.907s   5.59e-06s C 162214   17   
> GpuFromHost
>2.9%68.2%   0.700s   1.05e-05s C 667947   
> HostFromGpu
>2.6%70.7%   0.611s   6.40e-05s C 95421   
> GpuElemwise{Composite{(i0 + (-scalar_sigmoid(((i1 + i2) + i3}}[(0, 2)]
>2.1%72.9%   0.503s   5.28e-05s C 95421   
> GpuElemwise{Composite{((i0 * i1) - scalar_softplus(i1))},no_inplace}
>2.0%74.8%   0.468s   4.91e-05s C 95421   
> GpuElemwise{Composite{(i0 + (-scalar_sigmoid(i1)))}}[(0, 1)]
>1.9%76.7%   0.444s   1.16e-05s C 381684   
> GpuCAReduce{add}{0,1,1}
>1.7%78.4%   0.404s   4.24e-05s C 95421   
> GpuElemwise{Composite{((i0 + i1) + i2)}}[(0, 1)]
>1.4%79.8%   0.327s   3.43e-05s C 95421   
> GpuGemv{inplace}
>1.4%81.1%   0.322s   1.69e-05s C 190842   
> GpuCAReduce{add}{0,0,1}
>1.3%82.4%   0.313s   1.09e-05s C 286263   
> GpuElemwise{Composite{((i0 * i1) + i2)}}[(0, 2)]
>1.0%83.5%   0.246s   1.29e-05s C 190842   
> GpuElemwise{scalar_sigmoid,no_inplace}
>0.9%84.4%   0.221s   1.16e-06s C 190840   20   
> GpuReshape{3}
>0.9%85.3%   0.219s   1.15e-06s C 190840   20   
> GpuReshape{2}
>0.9%86.2%   0.214s   1.12e-05s C 190842   
> GpuElemwise{Composite{(i0 + (i1 * sqr(i2)))},no_inplace}
>... (remaining 49 Ops account for  13.76%(3.27s) of the runtime)
>
> Apply
> --
> <% time><#call>  
>   16.3%16.3%   3.882s   4.07e-04s   9542   165   
> GpuCAReduce{add}{1,1,1}(GpuElemwise{Composite{((i0 * i1) - 
> scalar_softplus(i1))},no_inplace}.0)
>3.4%19.7%   0.810s   8.48e-05s   9542   169   
> GpuElemwise{mul,no_inplace}(GpuDimShuffle{0,x,1,2}.0, CudaNdarrayConstant{
>3.4%23.1%   0.802s   8.40e-05s   

[theano-users] Re: AlexNet_theano_generate_toy_data.sh problems

[Jesse Livezey]

You can probably modify line 27 in make_labels.py to be
for ind in range(labels.size // batch_size):

This code was probably written with python 2 where division worked 
differently.

On Monday, March 13, 2017 at 8:45:39 AM UTC-7, Goffredo Giordano wrote:
>
> Hi,
> I'm a new user and I'm trying to study the ample world of machine 
> learning. I would like to run the theano_alexnet training from 
> https://github.com/uoguelph-mlrg/theano_alexnet.
> My computer is a Windows 10 native-machine 64 bit Intel core i7. I use 
> WinPython-64bit-3.4.4.4QT5 from WinPython 3.4.4.3, Visual Studio 2015 
> Community Edition Update 3, CUDA 8.0.44 (64-bit), cuDNN v5.1 (August 10, 
> 2016) for CUDA 8.0, Git source control based on MinGW compiler and OpenBLAS 
> 0.2.14. 
> As fundamental python libraries Theano is 0.9.0beta1 version, Scipy is 
> 0.19.0, Keras 1.2.2, Lasagne 0.2.dev1, Numpy 1.11.1, hickle 2.0.4, h5py 
> 2.6.0, pycuda, pylearn2, zeromq.
> I have downloaded the training images, the validation images and I have 
> unzipped the development kit from Imagenet dataset. I have configured the 
> paths.yaml with my folders but I do not know where I could find the val.txt 
> and train.txt files. I used the meta_clsloc.mat file and 
> ILSVRC2012_validation_ground_truth.txt file from the development kit from 
> Imagenet dataset. With the Git bash control i try to run the 
> generate_toy_data.sh and I can find the train_labels.npy, val_labels.npy, 
> img_mean.npy, shuffled_train_filenames.npy with the validation alex net 
> *.hkl files, but nothing in the training folder. Probably I forgot some 
> important features, so I would apologize previously. Thank you so much!
>
>
> Goffredo_Giordano@Goffredo MINGW64 /c/deep_learning/alexnet/preprocessing
> $ sh generate_toy_data.sh
> ciao
> generating toy dataset ...
>   make_hkl.py:72: 
> VisibleDeprecationWarning: using a non-integer number instead of an integer 
> will result in an error in the future
>   hkl.dump(img_batch[:, :, :, :half_size],
> make_hkl.py:76: VisibleDeprecationWarning: using a non-integer number 
> instead of an integer will result in an error in the future
>   hkl.dump(img_batch[:, :, :, half_size:],
> Traceback (most recent call last):
>   File "make_train_val_txt.py", line 26, in 
> synsets = scipy.io.loadmat(meta_clsloc_mat)['synsets'][0]
>   File 
> "C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\scipy\io\matlab\mio.py",
>  
> line 136, in loadmat
> matfile_dict = MR.get_variables(variable_names)
>   File 
> "C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\scipy\io\matlab\mio5.py",
>  
> line 272, in get_variables
> hdr, next_position = self.read_var_header()
>   File 
> "C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\scipy\io\matlab\mio5.py",
>  
> line 226, in read_var_header
> mdtype, byte_count = self._matrix_reader.read_full_tag()
>   File "scipy\io\matlab\mio5_utils.pyx", line 546, in 
> scipy.io.matlab.mio5_utils.VarReader5.read_full_tag 
> (scipy\io\matlab\mio5_utils.c:5330)
>   File "scipy\io\matlab\mio5_utils.pyx", line 554, in 
> scipy.io.matlab.mio5_utils.VarReader5.cread_full_tag 
> (scipy\io\matlab\mio5_utils.c:5400)
>   File "scipy\io\matlab\streams.pyx", line 164, in 
> scipy.io.matlab.streams.ZlibInputStream.read_into 
> (scipy\io\matlab\streams.c:3052)
>   File "scipy\io\matlab\streams.pyx", line 151, in 
> scipy.io.matlab.streams.ZlibInputStream._fill_buffer 
> (scipy\io\matlab\streams.c:2913)
> zlib.error: Error -3 while decompressing data: invalid distance too far 
> back
> make_labels.py:17: VisibleDeprecationWarning: using a non-integer number 
> instead of an integer will result in an error in the future
>   labels = labels[:labels.size / orig_batch_size * orig_batch_size]
> make_labels.py:23: VisibleDeprecationWarning: using a non-integer number 
> instead of an integer will result in an error in the future
>   labels_0 = labels.reshape((-1, batch_size))[::num_div].reshape(-1)
> make_labels.py:24: VisibleDeprecationWarning: using a non-integer number 
> instead of an integer will result in an error in the future
>   labels_1 = labels.reshape((-1, batch_size))[1::num_div].reshape(-1)
> Traceback (most recent call last):
>   File "make_labels.py", line 125, in 
> div_labels(train_label_name, orig_batch_size, num_div)
>   File "make_labels.py", line 27, in div_labels
> for ind in range(labels.size / batch_size):
> TypeError: 'float' object cannot be interpreted as an integer
>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Gradients are always 0 for custom loss function

[Jesse Livezey]

I'm not sure that such a list exists.

One heuristic is that if your function returns an int or boolean (binary) 
value, then the derivatives are probably going to be zero.

set_subtensor returns a modified tensor (potentially a float) and so the 
derivative with respect to the original tensor and new subtensor will 
generally be non-zero.

On Monday, March 6, 2017 at 3:20:45 PM UTC-8, 
tarom...@alum.northwestern.edu wrote:
>
> Thanks, is there a way to know what operations are allowed in the context 
> of building a loss function? I can see that T.eq would have 0 gradients 
> everywhere except the discontinuous point at which the function equals 1, 
> but I'm having trouble imagining what the gradient would be for something 
> like T.set_subtensor, which also seems to have a 0 gradient.
>
> On Monday, March 6, 2017 at 11:38:59 AM UTC-6, Jesse Livezey wrote:
>>
>> There is nothing wrong with using T.eq. But, the derivatives with respect 
>> to the inputs will be zero, so your cost function is not useful for 
>> training.
>>
>> On Sunday, March 5, 2017 at 8:03:12 PM UTC-8, 
>> tarom...@alum.northwestern.edu wrote:
>>>
>>> Thanks Jesse, so are there operations that are "safe" to use and others 
>>> that aren't? Where can I find this information? Also, I've used T.eq before 
>>> in another custom loss function which works correctly and doesn't return 0 
>>> gradients, but my use case there is in computing array indices, such as the 
>>> way I'm using it in this line:
>>>
>>> pred_up = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
>>> preprocess.up_index])
>>>
>>> Is T.eq ok to use in some contexts and not others?
>>>
>>> On Sunday, March 5, 2017 at 9:14:20 PM UTC-6, Jesse Livezey wrote:
>>>>
>>>> The gradient of T.eq will be zero (almost) everywhere and you're using 
>>>> it to compute num_win and num_lose.
>>>>
>>>> On Sunday, March 5, 2017 at 2:42:14 PM UTC-8, 
>>>> tarom...@alum.northwestern.edu wrote:
>>>>>
>>>>> Also, the return values of this loss function are small compared to 
>>>>> cross-entropy, some sample values after random initialization were around 
>>>>> +/- 0.01. There is a LSTM layer and the input sequences are thousands of 
>>>>> elements long, so I suspected vanishing gradients. However, I'm printing 
>>>>> out the min, max, and mean of the gradients w.r.t each parameter, and 
>>>>> they 
>>>>> are all exactly equal to 0, which seems to indicate a different problem.
>>>>>
>>>>> On Sunday, March 5, 2017 at 3:59:42 PM UTC-6, 
>>>>> tarom...@alum.northwestern.edu wrote:
>>>>>>
>>>>>> I have defined a custom loss function, and despite the loss function 
>>>>>> returning correct values given the inputs, the gradients are all always 
>>>>>> 0 
>>>>>> w.r.t each of my parameters. I am not suppressing any theano errors 
>>>>>> including the disconnected input error, so I can't explain what is 
>>>>>> causing 
>>>>>> this. I have copied the loss function below; in words, I first convert a 
>>>>>> 3 
>>>>>> class softmax output into a one hot representation, then I compare a 
>>>>>> subset 
>>>>>> of it to the response and compute a quantity of interest. More 
>>>>>> generally, I 
>>>>>> was under the impression that if one could express a function using 
>>>>>> theano 
>>>>>> ops, it could be used as a loss function. Is this not the case?
>>>>>>
>>>>>> def calc_one_hot_loss(pred, y, mask):
>>>>>> mask_flat = T.flatten(mask)
>>>>>> length = T.sum(mask_flat, dtype='int32')
>>>>>> pred_unmasked = pred[mask_flat.nonzero()]
>>>>>> max_indices = T.argmax(pred_unmasked, axis=1)
>>>>>> pred_zero = T.set_subtensor(pred_unmasked[:], 0)
>>>>>> pred_one_hot = T.set_subtensor(pred_zero[T.arange(length), 
>>>>>> max_indices], 1)
>>>>>> y_unmasked = y[mask_flat.nonzero()]
>>>>>> unchanged_col = pred_one_hot[:, preprocess.unchanged_index]
>>>>>> pred_up = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
>>>>>> preprocess.up_index])
>>>>>> pred_down = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
>>>>>> preprocess.down_index])
>>>>>> y_up = T.flatten(y_unmasked[T.eq(unchanged_col, 0).nonzero(), 
>>>>>> preprocess.up_index])
>>>>>> y_down = T.flatten(y_unmasked[T.eq(unchanged_col, 0).nonzero(), 
>>>>>> preprocess.down_index])
>>>>>> diff_up = T.abs_(pred_up - y_up)
>>>>>> diff_down = T.abs_(pred_down - y_down)
>>>>>> diff_sum = diff_up + diff_down
>>>>>> num_win = T.sum(T.eq(diff_sum, 0))
>>>>>> num_lose = T.sum(T.eq(diff_sum, 2))
>>>>>> loss = -1 * (num_win - num_lose) / length
>>>>>> return loss
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Gradients are always 0 for custom loss function

[Jesse Livezey]

There is nothing wrong with using T.eq. But, the derivatives with respect 
to the inputs will be zero, so your cost function is not useful for 
training.

On Sunday, March 5, 2017 at 8:03:12 PM UTC-8, 
tarom...@alum.northwestern.edu wrote:
>
> Thanks Jesse, so are there operations that are "safe" to use and others 
> that aren't? Where can I find this information? Also, I've used T.eq before 
> in another custom loss function which works correctly and doesn't return 0 
> gradients, but my use case there is in computing array indices, such as the 
> way I'm using it in this line:
>
> pred_up = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
> preprocess.up_index])
>
> Is T.eq ok to use in some contexts and not others?
>
> On Sunday, March 5, 2017 at 9:14:20 PM UTC-6, Jesse Livezey wrote:
>>
>> The gradient of T.eq will be zero (almost) everywhere and you're using it 
>> to compute num_win and num_lose.
>>
>> On Sunday, March 5, 2017 at 2:42:14 PM UTC-8, 
>> tarom...@alum.northwestern.edu wrote:
>>>
>>> Also, the return values of this loss function are small compared to 
>>> cross-entropy, some sample values after random initialization were around 
>>> +/- 0.01. There is a LSTM layer and the input sequences are thousands of 
>>> elements long, so I suspected vanishing gradients. However, I'm printing 
>>> out the min, max, and mean of the gradients w.r.t each parameter, and they 
>>> are all exactly equal to 0, which seems to indicate a different problem.
>>>
>>> On Sunday, March 5, 2017 at 3:59:42 PM UTC-6, 
>>> tarom...@alum.northwestern.edu wrote:
>>>>
>>>> I have defined a custom loss function, and despite the loss function 
>>>> returning correct values given the inputs, the gradients are all always 0 
>>>> w.r.t each of my parameters. I am not suppressing any theano errors 
>>>> including the disconnected input error, so I can't explain what is causing 
>>>> this. I have copied the loss function below; in words, I first convert a 3 
>>>> class softmax output into a one hot representation, then I compare a 
>>>> subset 
>>>> of it to the response and compute a quantity of interest. More generally, 
>>>> I 
>>>> was under the impression that if one could express a function using theano 
>>>> ops, it could be used as a loss function. Is this not the case?
>>>>
>>>> def calc_one_hot_loss(pred, y, mask):
>>>> mask_flat = T.flatten(mask)
>>>> length = T.sum(mask_flat, dtype='int32')
>>>> pred_unmasked = pred[mask_flat.nonzero()]
>>>> max_indices = T.argmax(pred_unmasked, axis=1)
>>>> pred_zero = T.set_subtensor(pred_unmasked[:], 0)
>>>> pred_one_hot = T.set_subtensor(pred_zero[T.arange(length), 
>>>> max_indices], 1)
>>>> y_unmasked = y[mask_flat.nonzero()]
>>>> unchanged_col = pred_one_hot[:, preprocess.unchanged_index]
>>>> pred_up = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
>>>> preprocess.up_index])
>>>> pred_down = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
>>>> preprocess.down_index])
>>>> y_up = T.flatten(y_unmasked[T.eq(unchanged_col, 0).nonzero(), 
>>>> preprocess.up_index])
>>>> y_down = T.flatten(y_unmasked[T.eq(unchanged_col, 0).nonzero(), 
>>>> preprocess.down_index])
>>>> diff_up = T.abs_(pred_up - y_up)
>>>> diff_down = T.abs_(pred_down - y_down)
>>>> diff_sum = diff_up + diff_down
>>>> num_win = T.sum(T.eq(diff_sum, 0))
>>>> num_lose = T.sum(T.eq(diff_sum, 2))
>>>> loss = -1 * (num_win - num_lose) / length
>>>> return loss
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Gradients are always 0 for custom loss function

[Jesse Livezey]

The gradient of T.eq will be zero (almost) everywhere and you're using it 
to compute num_win and num_lose.

On Sunday, March 5, 2017 at 2:42:14 PM UTC-8, 
tarom...@alum.northwestern.edu wrote:
>
> Also, the return values of this loss function are small compared to 
> cross-entropy, some sample values after random initialization were around 
> +/- 0.01. There is a LSTM layer and the input sequences are thousands of 
> elements long, so I suspected vanishing gradients. However, I'm printing 
> out the min, max, and mean of the gradients w.r.t each parameter, and they 
> are all exactly equal to 0, which seems to indicate a different problem.
>
> On Sunday, March 5, 2017 at 3:59:42 PM UTC-6, 
> tarom...@alum.northwestern.edu wrote:
>>
>> I have defined a custom loss function, and despite the loss function 
>> returning correct values given the inputs, the gradients are all always 0 
>> w.r.t each of my parameters. I am not suppressing any theano errors 
>> including the disconnected input error, so I can't explain what is causing 
>> this. I have copied the loss function below; in words, I first convert a 3 
>> class softmax output into a one hot representation, then I compare a subset 
>> of it to the response and compute a quantity of interest. More generally, I 
>> was under the impression that if one could express a function using theano 
>> ops, it could be used as a loss function. Is this not the case?
>>
>> def calc_one_hot_loss(pred, y, mask):
>> mask_flat = T.flatten(mask)
>> length = T.sum(mask_flat, dtype='int32')
>> pred_unmasked = pred[mask_flat.nonzero()]
>> max_indices = T.argmax(pred_unmasked, axis=1)
>> pred_zero = T.set_subtensor(pred_unmasked[:], 0)
>> pred_one_hot = T.set_subtensor(pred_zero[T.arange(length), max_indices], 
>> 1)
>> y_unmasked = y[mask_flat.nonzero()]
>> unchanged_col = pred_one_hot[:, preprocess.unchanged_index]
>> pred_up = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
>> preprocess.up_index])
>> pred_down = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
>> preprocess.down_index])
>> y_up = T.flatten(y_unmasked[T.eq(unchanged_col, 0).nonzero(), 
>> preprocess.up_index])
>> y_down = T.flatten(y_unmasked[T.eq(unchanged_col, 0).nonzero(), 
>> preprocess.down_index])
>> diff_up = T.abs_(pred_up - y_up)
>> diff_down = T.abs_(pred_down - y_down)
>> diff_sum = diff_up + diff_down
>> num_win = T.sum(T.eq(diff_sum, 0))
>> num_lose = T.sum(T.eq(diff_sum, 2))
>> loss = -1 * (num_win - num_lose) / length
>> return loss
>>
>>
>>
>>
>>
>>
>>
>>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Keeping word embeddings static

[Jesse Livezey]

If your model returns the embedding matrix in self.model.params here
https://github.com/attapol/nn_discourse_parser/blob/534a633d87d671126f135ccebfaa9817947730a7/nets/learning.py#L186
then it will be updated since everything returned in that line gets 
optimized. 

If you don't want it to be optimized you'll need to remove it from that 
list or remove it from self.sgs_updates and self.parameter_updates which 
get passed to your train function updates.

On Tuesday, February 21, 2017 at 5:39:35 AM UTC-8, Jimmy wrote:
>
> Hi,
>
> I have limited experience with Theano, but have found myself in the 
> position of modifying a feed forward neural network with an initial word 
> embedding layer. The goal is to keep the word embeddings static rather than 
> tuning their weights in the training process. Using Keras, this is simply 
> done by using the `trainable=False` parameter 
> 
>  
> when loading the embedding weights, but I'm unsure how to do this directly 
> in Theano. 
>
> The code I'm modifying can be found here 
> ,
>  
> and it is the AdagradTrainer class starting on line 175. I don't expect you 
> to analyze the code and come up with the solution, but I would very much 
> appreciate if you pointed me towards a minimal example on how updating 
> embedding weights and how to keep them static generally work.
>
> Thank you,
> Jimmy
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Logistic Regression for Semantic Segmentation

[Jesse Livezey]

The first line should probably be

logits = x.dot(w) + b.dimshuffle('x', 0')

On Monday, February 13, 2017 at 3:05:34 PM UTC-8, Jesse Livezey wrote:
>
> You haven't used w or b in the graph to compute the cost function. You 
> need to add something like
>
> logits = w.dot(x) + b.dimshuffle('x', 0')
> logitsdev = logits - logits.max(1, keepdims=True)
>
> you probably also want b to be of length 257 rather than a scalar.
>
> On Monday, February 13, 2017 at 9:06:57 AM UTC-8, Dimitris Marm wrote:
>>
>>
>> Hello,
>>
>> I am new to Theano and I have seen that actually is not straight-forward 
>> how to apply semantic segmentation over multiclass and multilabel labels.
>>
>> Particularly the problem I seen many people face is regarding the 
>> logistic-regression layer in the end of the network. For tackling this 
>> problem I constructed a single 
>>
>> *logistic-regression layer for 2d labels.*I guess my code still has some 
>> bug but I am unable to find  it !
>>
>>
>> The code 
>>
>> =
>>
>> import numpy as np
>> import random
>> from theano import shared
>> from theano import tensor as T
>> import theano
>>
>> #import pdb; pdb.set_trace()
>>
>> theano.config.optimizer='fast_compile'
>> theano.config.exception_verbosity='high'
>> theano.config.compute_test_value = 'warn'
>>
>> features = []
>> labels = []
>>
>>
>> # === theano =
>>
>> for i in range(10):
>>
>> if i == 0:
>> 
>> features = np.array([random.randint(0,256) for t in range(0,128*
>> 128)]).reshape((1, 1, 128,128))
>> labels = np.array([random.randint(0,256) for t in range(0,128*128
>> )]).reshape((1, 1, 128,128))
>>
>> else:
>>
>> features= np.append(features, np.array([random.randint(0,256) for 
>> t in range(0,128*128)]).reshape((1, 1, 128,128)),axis= 0)
>>
>> labels= np.append(features, np.array([random.randint(0,256) for 
>> t in range(0,128*128)]).reshape((1, 1, 128,128)),axis= 0)
>>
>> # Loss
>>
>> def train_lr(x_train, y_train, regu_coeff = 0.00015, step = 0.001):
>>
>> x = T.matrix("x")
>> y = T.matrix("y")
>> 
>> w = theano.shared(
>> value = np.random.randn(128*128,257),
>> name='w', 
>> borrow=True
>> )
>> 
>> b = theano.shared(0., name='b')
>> 
>> # 2d-softmax implementation
>> xdev = x - x.max(1, keepdims=True)
>> log_pred = xdev - T.log(T.sum(T.exp(xdev), axis=1, keepdims=True))
>>
>> cost = T.mean(-T.sum(y * log_pred, axis=1)) # cross entropy 
>> 
>> gw, gb = T.grad(cost, [w, b]) # gradient
>>
>> train = theano.function(
>> inputs=[x,y],
>> outputs=[log_pred],
>> updates=((w, w - step * gw), (b, b - step * gb)))
>>
>> for i in range(100):
>> train(x_train,y_train)
>>
>> return w, b
>>
>> pp = train_lr(features, labels)
>>
>>
>>
>> The error I am getting
>>
>> ===
>>
>>
>>
>>
>>
>>
>>
>> *theano.gradient.DisconnectedInputError: grad method was asked to compute 
>> the gradient with respect to a variable that is not part of the 
>> computational graph of the cost, or is used only by a non-differentiable 
>> operator: w~*I really do not know how to proceed as I don't 
>> understand why the w (weights) are not differentiable.
>>
>> I would appreciate any hints 
>>
>> Regards
>>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Logistic Regression for Semantic Segmentation

[Jesse Livezey]

You haven't used w or b in the graph to compute the cost function. You need 
to add something like

logits = w.dot(x) + b.dimshuffle('x', 0')
logitsdev = logits - logits.max(1, keepdims=True)

you probably also want b to be of length 257 rather than a scalar.

On Monday, February 13, 2017 at 9:06:57 AM UTC-8, Dimitris Marm wrote:
>
>
> Hello,
>
> I am new to Theano and I have seen that actually is not straight-forward 
> how to apply semantic segmentation over multiclass and multilabel labels.
>
> Particularly the problem I seen many people face is regarding the 
> logistic-regression layer in the end of the network. For tackling this 
> problem I constructed a single 
>
> *logistic-regression layer for 2d labels.*I guess my code still has some 
> bug but I am unable to find  it !
>
>
> The code 
>
> =
>
> import numpy as np
> import random
> from theano import shared
> from theano import tensor as T
> import theano
>
> #import pdb; pdb.set_trace()
>
> theano.config.optimizer='fast_compile'
> theano.config.exception_verbosity='high'
> theano.config.compute_test_value = 'warn'
>
> features = []
> labels = []
>
>
> # === theano =
>
> for i in range(10):
>
> if i == 0:
> 
> features = np.array([random.randint(0,256) for t in range(0,128*
> 128)]).reshape((1, 1, 128,128))
> labels = np.array([random.randint(0,256) for t in range(0,128*128
> )]).reshape((1, 1, 128,128))
>
> else:
>
> features= np.append(features, np.array([random.randint(0,256) for 
> t in range(0,128*128)]).reshape((1, 1, 128,128)),axis= 0)
>
> labels= np.append(features, np.array([random.randint(0,256) for t 
> in range(0,128*128)]).reshape((1, 1, 128,128)),axis= 0)
>
> # Loss
>
> def train_lr(x_train, y_train, regu_coeff = 0.00015, step = 0.001):
>
> x = T.matrix("x")
> y = T.matrix("y")
> 
> w = theano.shared(
> value = np.random.randn(128*128,257),
> name='w', 
> borrow=True
> )
> 
> b = theano.shared(0., name='b')
> 
> # 2d-softmax implementation
> xdev = x - x.max(1, keepdims=True)
> log_pred = xdev - T.log(T.sum(T.exp(xdev), axis=1, keepdims=True))
>
> cost = T.mean(-T.sum(y * log_pred, axis=1)) # cross entropy 
> 
> gw, gb = T.grad(cost, [w, b]) # gradient
>
> train = theano.function(
> inputs=[x,y],
> outputs=[log_pred],
> updates=((w, w - step * gw), (b, b - step * gb)))
>
> for i in range(100):
> train(x_train,y_train)
>
> return w, b
>
> pp = train_lr(features, labels)
>
>
>
> The error I am getting
>
> ===
>
>
>
>
>
>
>
> *theano.gradient.DisconnectedInputError: grad method was asked to compute 
> the gradient with respect to a variable that is not part of the 
> computational graph of the cost, or is used only by a non-differentiable 
> operator: w~*I really do not know how to proceed as I don't 
> understand why the w (weights) are not differentiable.
>
> I would appreciate any hints 
>
> Regards
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Re: How to get the value of theano tensor variable

[Jesse Livezey]

Did it give an error? If you're debugging things, eval is the right
function to use. If you're outputting the value of these variables during
training, you can just add them to the list of outputs when you create the
theano function.

On Fri, Jan 27, 2017 at 8:49 PM chathu matharage <chathui...@gmail.com>
wrote:

> I need to get the value of Elementwise{tanh,no_inplace}.0 variable which
> is the output of error of logistic regression layer in convolution neural
> network. I tried with eval(). But it didn't work. Is there a method to get
> the error value of this type of variable?
>
>
> On Saturday, January 28, 2017 at 6:27:24 AM UTC+5:30, Jesse Livezey wrote:
>
> You can use eval
>
> http://deeplearning.net/software/theano/library/gof/graph.html#theano.gof.graph.Variable.eval
>
> On Friday, January 27, 2017 at 4:39:56 PM UTC-8, chathu matharage wrote:
>
> Hi,
> Is there a method to print the value of a tensor variable?
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "theano-users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/theano-users/WVcBsiXG2w0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> theano-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: How to get the value of theano tensor variable

[Jesse Livezey]

You can use eval
http://deeplearning.net/software/theano/library/gof/graph.html#theano.gof.graph.Variable.eval

On Friday, January 27, 2017 at 4:39:56 PM UTC-8, chathu matharage wrote:
>
> Hi,
> Is there a method to print the value of a tensor variable?
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: How should I concatenate the matrices in a tensor?

[Jesse Livezey]

You can use reshape rather than concatenate (similar to how you could do it 
in numpy).

On Saturday, December 24, 2016 at 2:48:58 AM UTC-5, Quanty wrote:
>
> I know there is a T.concatenate function but the input of it can only be 
> list. But the output of theano.scan is a tensor. For example, I want to 
> make a tensor with the shape (2,3,4) which is the output of theano.scan to 
> be a matrix of (2,12), I can't do it with T.concatenate because it is not a 
> list of (3,4) matrices.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Concurrent GPU threads in Theano

[Jesse Livezey]

If you can create a theano vector that has all of the i's and a second 
theano vector that has all of the j's, then you can just do i*j and will 
will perform all of the multiplications in parallel.

On Wednesday, October 26, 2016 at 11:48:06 PM UTC-7, kd...@cornell.edu 
wrote:
>
> I would like to compute the result of i*j for a number of i's and j's, and 
> I would like to do so concurrently. If I use the scan function over my 
> sequence of i's and j's, I will get my desired result, but it will not 
> perform the operations concurrently. If I have 100 cores in my single GPU, 
> I would like there to be 100 asynchronous computations (technically more 
> since each core has multiple threads) of the multiplication and final 
> assignment to one vector that will be returned. This is similar to how 
> multiprocessing works in base python with CPU cores. The Theano tutorial 
> claims that it uses GPU asynchronous capabilities, but I am not sure of 
> that as I have ran scan functions, and they seems to go as fast or slower 
> than the CPU.
>
> Should I not use scan? Can this even be done in Theano? Do I have to use 
> PyCUDA?
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Can't import Theano!!! configparser.DuplicateSectionError: While reading from '/home/rik/.theanorc' [line 6]: section 'nvcc' already exists

[Jesse Livezey]

What are the contents of this file?
/home/rik/.theanorc
Does is have [nvcc] multiple times?

On Monday, October 10, 2016 at 9:00:52 PM UTC-7, Rik wrote:
>
> Hey all,
>
> Python won't let me import theano. Here is the error I get:
>
> >>> import theano
> Traceback (most recent call last):
>   File "", line 1, in 
>   File 
> "/home/rik/anaconda3/lib/python3.5/site-packages/theano/__init__.py", line 
> 39, in 
> from theano.configdefaults import config
>   File 
> "/home/rik/anaconda3/lib/python3.5/site-packages/theano/configdefaults.py", 
> line 6, in 
> from theano.configparser import (AddConfigVar, BoolParam, ConfigParam, 
> EnumStr,
>   File 
> "/home/rik/anaconda3/lib/python3.5/site-packages/theano/configparser.py", 
> line 83, in 
> theano_cfg.read(config_files)
>   File "/home/rik/anaconda3/lib/python3.5/configparser.py", line 696, in 
> read
> self._read(fp, filename)
>   File "/home/rik/anaconda3/lib/python3.5/configparser.py", line 1063, in 
> _read
> lineno)
> configparser.DuplicateSectionError: While reading from 
> '/home/rik/.theanorc' [line  6]: section 'nvcc' already exists
>
> I think it has to do with tensorflow being installed since nvcc is a CUDA 
> thing I believe. not sure what to do to fix this. PLEASE HELP!!!
>
> -Rik
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Input error(extra dimensions) at theano.function

[Jesse Livezey]

Your inputs should be tensor4 rather than matrix if you're passing them 
into a CNN.

On Tuesday, September 6, 2016 at 7:22:38 AM UTC-7, Ganesh Iyer wrote:
>
>  
> Hi guys, 
>
> I'm new to this group and theano in general. I'm trying to send 2 image 
> patches, greyscale ( 2D numpy arrays of size (9,9) using cv2.imread(name,0) 
> ) through a CNN architecture. I'm giving these as inputs to theano.function.
>
> train_set_left=np.float64(train_set_left)
> train_set_right_positive=np.float64(train_set_right_positive)
>
> train_model=theano.function(inputs=[input_left,input_right] 
> ,outputs=[s_plus])
> print(train_model(train_set_left,train_set_right_positive))
>
> The error I get at this point is:
>
> at index 0(0-based)', 'Wrong number of dimensions: expected 4, got 2 with 
> shape (9, 9).')
>
>
> input_left and input_right are defined earlier in the code as:
>
> input_left=T.dmatrix('input_left')
> input_right=T.dmatrix('input_right')
>
> Is there something wrong with the input dimensions in this case?
>
> Full Code: http://pastebin.com/33fTyb3K
> The code itself is based on the LeNet tutorial, but is a bit messy. 
>
> Please Help.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: error 4D input data

[Jesse Livezey]

One of your labels is too large, or possibly too small. Are your labels 
from 0 to n-1 or 1 to n? They should be 0 to n-1.

On Tuesday, September 6, 2016 at 2:19:18 AM UTC-7, Beatriz G. wrote:
>
> HI everyone
>
> I am trying to use 4 dimension image, but I get the following error and I 
> do not know what it means:
>
> ValueError: y_i value out of bounds
> Apply node that caused the error: 
> CrossentropySoftmaxArgmax1HotWithBias(Dot22.0, b, Subtensor{int64:int64:}.0)
> Toposort index: 34
> Inputs types: [TensorType(float64, matrix), TensorType(float64, vector), 
> TensorType(int32, vector)]
> Inputs shapes: [(20, 4), (4,), (20,)]
> Inputs strides: [(32, 8), (8,), (4,)]
> Inputs values: ['not shown', array([ 0.,  0.,  0.,  0.]), 'not shown']
> Outputs clients: 
> [[Sum{acc_dtype=float64}(CrossentropySoftmaxArgmax1HotWithBias.0)], 
> [CrossentropySoftmax1HotWithBiasDx(Elemwise{Inv}[(0, 0)].0, 
> CrossentropySoftmaxArgmax1HotWithBias.1, Subtensor{int64:int64:}.0)], []]
>
> Backtrace when the node is created(use Theano flag traceback.limit=N to 
> make it longer):
>   File "/home/beaa/Escritorio/Theano/Separando_Lenet.py", line 446, in 
> 
> evaluate_lenet5()
>   File "/home/beaa/Escritorio/Theano/Separando_Lenet.py", line 257, in 
> evaluate_lenet5
> cost = layer3.negative_log_likelihood(y)
>   File "/home/beaa/Escritorio/Theano/logistic_sgd.py", line 146, in 
> negative_log_likelihood
> return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]), y])
>
> HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and 
> storage map footprint of this apply node.
>
>
> Here is how I give the data to the layers:
>
>
> layer0 = LeNetConvPoolLayer(
> rng,
> input=layer0_input,
> image_shape=(batch_size, 4, 104, 52),
> filter_shape=(nkerns[0], 4, 5, 5),
> poolsize=(2, 2)
> )
>
>
> layer1 = LeNetConvPoolLayer(
> rng,
> input=layer0.output,
> image_shape=(batch_size, nkerns[0], 50, 24),
> filter_shape=(nkerns[1], nkerns[0], 5, 5),
> poolsize=(2, 2)
>
>
>
> My data is 104*52*4.
>
>
> Thanks in advance. Regards.
>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] TypeError: unhashable type: 'numpy.ndarray'

[Jesse Livezey]

@Fred and Pascal, I think Beatriz's question was answered in another thread.

On Wednesday, August 31, 2016 at 12:58:11 PM UTC-7, Pascal Lamblin wrote:
>
> `salidas_capa3` is a theano function, which is a callable object. 
> However, you are trying to _index_ into it using 
> `salidas_capa3[test_model(i)]`. 
> What is the behaviour you would expect from that code? 
>
> On Wed, Jul 27, 2016, Beatriz G. wrote: 
> > Hi guays, 
> > 
> > I am having a similar problem, I am trying to compile the following 
> theano 
> > function: 
> > 
> > layer3 = LogisticRegression(input=layer2.output, n_in=100, n_out=4) 
> > 
> > 
> > salidas_capa3 = theano.function( 
> > [index], 
> > layer3.y_pred, 
> > givens={ 
> > x: test_set_x[index * batch_size: (index + 1) * batch_size] 
> > } 
> > ) 
> > 
> > 
> > 
> > Which I call in the test part: 
> > 
> > 
> > for i in range(n_test_batches): 
> > test_losses = [test_model(i)] 
> > y_pred_test = salidas_capa3[test_model(i)] 
> > print y_pred_test 
> > test_score = numpy.mean(test_losses) 
> > 
> > 
> > 
> > And I am getting the next error: 
> > 
> > 
> > Traceback (most recent call last): 
> >   File "/home/beaa/Escritorio/Theano/Separando_Lenet.py", line 416, in 
>  
> > evaluate_lenet5() 
> >   File "/home/beaa/Escritorio/Theano/Separando_Lenet.py", line 392, in 
> evaluate_lenet5 
> > y_pred_test = salidas_capa3[test_model(i)] 
> >   File 
> "/home/beaa/.local/lib/python2.7/site-packages/theano/compile/function_module.py",
>  
> line 545, in __getitem__ 
> > return self.value[item] 
> >   File 
> "/home/beaa/.local/lib/python2.7/site-packages/theano/compile/function_module.py",
>  
> line 480, in __getitem__ 
> > s = finder[item] 
> > TypeError: unhashable type: 'numpy.ndarray' 
> > 
> > 
> > 
> > 
> > Anyone could guide me? 
> > 
> > 
> > Here is my code: 
> > 
> > 
> > class LeNetConvPoolLayer(object): 
> > """Pool Layer of a convolutional network """ 
> > 
> > def __init__(self, rng, input, filter_shape, image_shape, 
> poolsize=(2, 2)): 
> > """ 
> > Allocate a LeNetConvPoolLayer with shared variable internal 
> parameters. 
> > :type rng: numpy.random.RandomState 
> > :param rng: a random number generator used to initialize weights 
> > :type input: theano.tensor.dtensor4 
> > :param input: symbolic image tensor, of shape image_shape 
> > :type filter_shape: tuple or list of length 4 
> > :param filter_shape: (number of filters, num input feature maps, 
> >   filter height, filter width) 
> > :type image_shape: tuple or list of length 4 
> > :param image_shape: (batch size, num input feature maps, 
> >  image height, image width) 
> > :type poolsize: tuple or list of length 2 
> > :param poolsize: the downsampling (pooling) factor (#rows, 
> #cols) 
> > """ 
> > 
> > assert image_shape[1] == filter_shape[1] 
>#El numero de feature maps sea igual en ambas variables 
> > self.input = input 
> > 
> > # there are "num input feature maps * filter height * filter 
> width" 
> > # inputs to each hidden unit 
> > fan_in = numpy.prod(filter_shape[1:])   
>  #el numero de neuronas en la capa anterio 
> > # each unit in the lower layer receives a gradient from: 
> > # "num output feature maps * filter height * filter width" / 
> > #   pooling size 
> > 
> > fan_out = (filter_shape[0] * numpy.prod(filter_shape[2:]) / 
>  #El numero de neuronas de salida: numero de filtros*alto*ancho del 
> filtro/poolsize(tam*tam) 
> >numpy.prod(poolsize)) 
> > # initialize weights with random weights 
> > 
> > # initialize weights with random weights 
> > W_bound = numpy.sqrt(6. / (fan_in + fan_out)) 
> > self.W = theano.shared( 
> > numpy.asarray( 
> > rng.uniform(low=-W_bound, high=W_bound, 
> size=filter_shape), 
> > # Se calcula asi el W_bound por la funcion  de 
> activacion tangencial. 
> > # Los pesos dependen del tamanyo del filtro(neuronas) 
> > 
> > dtype=theano.config.floatX  # Para que sea valido en gpu 
> > ), 
> > borrow=True 
> > ) 
> > 
> > # the bias is a 1D tensor -- one bias per output feature map 
> > b_values = numpy.zeros((filter_shape[0],), 
> dtype=theano.config.floatX) 
> > self.b = theano.shared(value=b_values, borrow=True) 
> > 
> > # convolve input feature maps with filters 
> > conv_out = conv.conv2d( 
> > input=input, 
> > filters=self.W, 
> > filter_shape=filter_shape, 
> > image_shape=image_shape 
> > ) 
> > 
> > # downsample each feature map individually, 

[theano-users] Re: get test labels

[Jesse Livezey]

I just changed

salidas_capa3[test_model(i)]

to

salidas_capa3(i)


the function salidas_capa3 expects a batch index as an argument.

On Sunday, July 31, 2016 at 3:16:45 PM UTC-4, Beatriz G. wrote:
>
> Is it not what I have given to salidas_capa3?
>
> I am really thankful for your help, really, really thankful.
>
>
> El viernes, 29 de julio de 2016, 4:00:51 (UTC+2), Jesse Livezey escribió:
>>
>> I think you just want to do
>>
>> for i in range(n_test_batches):
>> test_losses = [test_model(i)]
>> y_pred_test = salidas_capa3(i)
>> print y_pred_test
>>
>>
>> The salidas_capa3 function expects a minibatch index as an argument.
>>
>> On Wednesday, July 27, 2016 at 11:27:08 PM UTC-7, Beatriz G. wrote:
>>>
>>> I am not able of extract the value of that function at that point, I 
>>> have debugged and I I have gotten the results of test_model in the attached 
>>> pic.
>>>
>>> Thank you for your help.
>>>
>>>
>>>
>>> What is the value of test_model(i) at that point? I think it should be 
>>>> an array of indices.
>>>>
>>>> On Wednesday, July 27, 2016 at 1:52:27 AM UTC-7, Beatriz G. wrote:
>>>>>
>>>>> Hi Jesse, thank you for your reply.
>>>>>
>>>>> I have tried to use it when I test:
>>>>>
>>>>> #Aqui se tiene que cargar la red
>>>>>
>>>>> layer0.W.set_value(w0_test)
>>>>> layer0.b.set_value(b0_test)
>>>>>
>>>>> layer1.W.set_value(w1_test)
>>>>> layer1.b.set_value(b1_test)
>>>>>
>>>>> layer2.W.set_value(w2_test)
>>>>> layer2.b.set_value(b2_test)
>>>>>
>>>>> # test it on the test set
>>>>> for i in range(n_test_batches):
>>>>> test_losses = [test_model(i)]
>>>>> y_pred_test = salidas_capa3[test_model(i)]
>>>>> print y_pred_test
>>>>> test_score = numpy.mean(test_losses)
>>>>>
>>>>> print((' test error of best model %f %%') % (test_score * 100.))
>>>>>
>>>>>
>>>>>
>>>>> but I get the following error:
>>>>>
>>>>>
>>>>> Traceback (most recent call last):
>>>>>   File "/home/beaa/Escritorio/Theano/Separando_Lenet.py", line 414, in 
>>>>> 
>>>>> evaluate_lenet5()
>>>>>   File "/home/beaa/Escritorio/Theano/Separando_Lenet.py", line 390, in 
>>>>> evaluate_lenet5
>>>>> y_pred_test = salidas_capa3[test_model(i)]
>>>>>   File 
>>>>> "/home/beaa/.local/lib/python2.7/site-packages/theano/compile/function_module.py",
>>>>>  line 545, in __getitem__
>>>>> return self.value[item]
>>>>>   File 
>>>>> "/home/beaa/.local/lib/python2.7/site-packages/theano/compile/function_module.py",
>>>>>  line 480, in __getitem__
>>>>> s = finder[item]
>>>>> TypeError: unhashable type: 'numpy.ndarray'
>>>>>
>>>>>
>>>>>
>>>>> and I do not know what produces it.
>>>>>
>>>>>
>>>>> Regards
>>>>>
>>>>>
>>>>> El miércoles, 27 de julio de 2016, 2:29:24 (UTC+2), Jesse Livezey 
>>>>> escribió:
>>>>>>
>>>>>> You should be able to use this function to output y_pred
>>>>>>
>>>>>> salidas_capa3 = theano.function(
>>>>>> [index],
>>>>>> layer3.y_pred,
>>>>>> givens={
>>>>>> x: test_set_x[index * batch_size: (index + 1) * batch_size],
>>>>>> }
>>>>>> )
>>>>>>
>>>>>>
>>>>>> On Monday, July 25, 2016 at 3:09:09 AM UTC-7, Beatriz G. wrote:
>>>>>>>
>>>>>>> Hi, anyone knows how to get the test labels that the classifier has 
>>>>>>> given to the data? 
>>>>>>>
>>>>>>> I would like to extrat the data that has not been well classified.
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: get test labels

[Jesse Livezey]

You should be able to use this function to output y_pred

salidas_capa3 = theano.function(
[index],
layer3.y_pred,
givens={
x: test_set_x[index * batch_size: (index + 1) * batch_size],
}
)


On Monday, July 25, 2016 at 3:09:09 AM UTC-7, Beatriz G. wrote:
>
> Hi, anyone knows how to get the test labels that the classifier has given 
> to the data? 
>
> I would like to extrat the data that has not been well classified.
>
> Regards.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Non-linear convolution

[Jesse Livezey]

You could do the L2 norm using the regular convolution by using the identity
||w-x||^2 = ||w||^2-2||w dot x||+||x||^2

and using
http://deeplearning.net/software/theano/library/tensor/nnet/neighbours.html#theano.tensor.nnet.neighbours.images2neibs
to help with the ||x||^2
and regular convolution to compute the
||w dot x||
term.

Other nonlinearities might be more tricky.

On Monday, July 25, 2016 at 12:08:56 PM UTC-7, Geppetto Null wrote:
>
> Hi everyone,
>
> I would like to modify the 2D convolution in order to introduce some 
> non-linear operation. For instance, instead of performing w^Tx+b = 
> dot(w,x)+b in each receptive field (i.e., patch of the input --as is by 
> default), I would like to perform the operation ||w-x||^2+b to each 
> receptive field, or some other non-linear operation (some other norm, for 
> example L-1). Is this doable in Theano/Lasagne?
>
> Thank you very much in advance.
> Best,
> Christos
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.