class BatchedDiag(tt.Op):
"""
Fast BatchedDiag allocation
"""
__props__ = ()
def make_node(self, diag):
diag = tt.as_tensor_variable(diag)
if diag.type.ndim != 2:
raise TypeError('data argument must be a matrix', diag.type)
return tt.Apply(self, [diag], [tt.tensor3(dtype=diag.dtype)])
def perform(self, node, ins, outs, params=None):
(C,) = ins
(z,) = outs
bc = C.shape[0]
dim = C.shape[-1]
Cd = np.zeros((bc, dim, dim), C.dtype)
bidx = np.repeat(np.arange(bc), dim)
didx = np.tile(np.arange(dim), bc)
Cd[bidx, didx, didx] = C.flatten()
z[0] = Cd
def grad(self, inputs, gout):
(gz,) = gout
idx = tt.arange(gz.shape[-1])
return [gz[..., idx, idx]]
def infer_shape(self, nodes, shapes):
return [(shapes[0][0], ) + (shapes[0][1],) * 2]
Here is code for Custom Op that might work faster when taking gradients
суббота, 7 мая 2016 г., 16:00:54 UTC+3 пользователь Tambet Matiisen написал:
>
> OK, solved. I used Keras wrapper K.zeros(), but this created Numpy matrix
> of zeros, which failed with Theano expression as dimension. After switching
> to full Theano implementation the error went away. The final code looks
> like this:
>
> # initialize with zeros
> batch_size = x.shape[0]
> a = T.zeros((batch_size, num_actuators, num_actuators))
> # set diagonal elements
> batch_idx = T.extra_ops.repeat(T.arange(batch_size), num_actuators)
> diag_idx = T.tile(T.arange(num_actuators), batch_size)
> b = T.set_subtensor(a[batch_idx, diag_idx, diag_idx],
> T.flatten(T.exp(x[:, :num_actuators])))
> # set lower triangle
> cols = np.concatenate([np.array(range(i), dtype=np.uint) for i in
> xrange(num_actuators)])
> rows = np.concatenate([np.array([i]*i, dtype=np.uint) for i in
> xrange(num_actuators)])
> cols_idx = T.tile(T.as_tensor_variable(cols), batch_size)
> rows_idx = T.tile(T.as_tensor_variable(rows), batch_size)
> batch_idx = T.extra_ops.repeat(T.arange(batch_size), len(cols))
> c = T.set_subtensor(b[batch_idx, rows_idx, cols_idx], T.flatten(x[:,
> num_actuators:]))
>
> Thanks injecting me belief that it is possible!
>
> Tambet
>
> reede, 6. mai 2016 17:57.02 UTC+3 kirjutas nouiz:
>>
>> what error do you get?
>>
>>
>> On Fri, May 6, 2016 at 10:54 AM, Tambet Matiisen <[email protected]>
>> wrote:
>>
>>> I could not figure out how make broadcasting work here, so I implemented
>>> option 2.
>>>
>>> num_actuators=4
>>> x = K.variable([range(num_actuators*(num_actuators+1)/2)]*5)
>>>
>>> batch_size = K.shape(x)[0]
>>> a = K.zeros((batch_size.eval(), num_actuators, num_actuators))
>>>
>>> # populate diagonal
>>> batch_idx = T.extra_ops.repeat(T.arange(batch_size), num_actuators)
>>> diag_idx = T.tile(T.arange(num_actuators), batch_size)
>>> b = T.set_subtensor(a[batch_idx, diag_idx, diag_idx],
>>> T.flatten(K.exp(x[:, :num_actuators])))
>>>
>>> # populate lower triangle
>>> cols = np.concatenate([np.array(range(i), dtype=np.uint) for i in
>>> xrange(num_actuators)])
>>> rows = np.concatenate([np.array([i]*i, dtype=np.uint) for i in
>>> xrange(num_actuators)])
>>> cols_idx = T.tile(K.variable(cols, dtype=int), batch_size)
>>> rows_idx = T.tile(K.variable(rows, dtype=int), batch_size)
>>> batch_idx = T.extra_ops.repeat(T.arange(batch_size), len(cols))
>>> c = T.set_subtensor(b[batch_idx, rows_idx, cols_idx], T.flatten(x[:,
>>> num_actuators:]))
>>>
>>> It works nicely, but only because I eval() batch_size when creating all
>>> zeros array. In real application I don't know the batch size beforehand and
>>> using it without eval() gives an error. So the question is - can you create
>>> a matrix in Theano dynamically, depending on some value in computational
>>> graph?
>>>
>>> Tambet
>>>
>>> reede, 6. mai 2016 16:14.59 UTC+3 kirjutas nouiz:
>>>>
>>>> broadcasting could be in theory more efficient. So this would request
>>>> that you try option 1.
>>>>
>>>> Otherwise, both should work.
>>>>
>>>> Fred
>>>>
>>>> On Fri, May 6, 2016 at 9:12 AM, Tambet Matiisen <[email protected]>
>>>> wrote:
>>>>
>>>>> Actually I know the dimensions of the matrix beforehand, so I can do
>>>>> those calculations in Python+Numpy. Following seems to do the trick:
>>>>>
>>>>> num_actuators = 3
>>>>> x = [1,2,3,4,5,6]
>>>>> a = K.zeros((num_actuators, num_actuators))
>>>>>
>>>>> # set diagonal elements
>>>>> b = T.set_subtensor(a[range(num_actuators), range(num_actuators)],
>>>>> K.exp(x[:num_actuators]))
>>>>>
>>>>> # set lower triangle
>>>>> cols = np.concatenate([np.array(range(i), dtype=np.uint) for i in
>>>>> xrange(num_actuators)])
>>>>> rows = np.concatenate([np.array([i]*i, dtype=np.uint) for i in
>>>>> xrange(num_actuators)])
>>>>> c = T.set_subtensor(b[rows, cols], x[num_actuators:])
>>>>>
>>>>> K.eval(c)
>>>>>
>>>>>
>>>>> array([[ 2.71828175, 0. , 0. ],
>>>>> [ 4. , 7.38905621, 0. ],
>>>>> [ 5. , 6. , 20.08553696]], dtype=float32)
>>>>>
>>>>>
>>>>> (I'm mixing Keras and Theano functions here, but I guess you
>>>>> understand the idea.)
>>>>>
>>>>> Now the problem is the following - actually x is not 1D, but 2D, the
>>>>> first dimension is batch size. So I would like to kind of broadcast this
>>>>> operation over first dimension of x. Is there any way to do it?
>>>>>
>>>>> An alternative would be to
>>>>> 1. construct a to be 3D, first dimension batch size,
>>>>> 2. repeat all index ranges batch size times.
>>>>> Sounds quite inefficient, but I guess doable.
>>>>>
>>>>> Tambet
>>>>>
>>>>> reede, 6. mai 2016 1:25.02 UTC+3 kirjutas nouiz:
>>>>>
>>>>>>
>>>>>> Le 5 mai 2016 16:18, "Tambet Matiisen" <[email protected]> a
>>>>>> écrit :
>>>>>> >
>>>>>> > Thanks Fred for a hint! Following seems to work (I'm using K is the
>>>>>> Keras equivalent of theano.tensor):
>>>>>> >
>>>>>> > a = K.zeros((3,3))
>>>>>> > K.eval(a)
>>>>>> >
>>>>>> > array([[ 0., 0., 0.],
>>>>>> > [ 0., 0., 0.],
>>>>>> > [ 0., 0., 0.]], dtype=float32)
>>>>>> >
>>>>>> > <br><br><br>
>>>>>> >
>>>>>> > b = T.set_subtensor(a[[0,1,2],[0,1,2]], [1,2,3])
>>>>>> >
>>>>>> > K.eval(b)
>>>>>> >
>>>>>> >
>>>>>> > array([[ 1., 0., 0.],
>>>>>> > [ 0., 2., 0.],
>>>>>> > [ 0., 0., 3.]], dtype=float32)
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > But what if I don't know the matrix dimensions beforehand? Can I
>>>>>> produce the list of indexes to assign using just Theano arithmetics?
>>>>>>
>>>>>> Yes. There is T.a range(...) That you can use.
>>>>>>
>>>>>> Fred
>>>>>> >
>>>>>> > Tambet
>>>>>> >
>>>>>> >
>>>>>> > neljapäev, 5. mai 2016 17:30.08 UTC+3 kirjutas nouiz:
>>>>>> >>
>>>>>> >> The first idea I have is to init a vector of zeros of 9 element
>>>>>> and use set_subtensor to set the indices to the value you want. Then
>>>>>> reshape to a matrix.
>>>>>> >>
>>>>>> >> Fred
>>>>>> >>
>>>>>> >> On Thu, May 5, 2016 at 5:07 AM, Tambet Matiisen <
>>>>>> [email protected]> wrote:
>>>>>> >>>
>>>>>> >>> Hi everyone!
>>>>>> >>>
>>>>>> >>> I'm trying to apply T.diag() and T.tril() operations over batch
>>>>>> of matrices, so that first dimension is preserved. Theano doesn't seem
>>>>>> to
>>>>>> provide built-in function to do that. Is there any other way to achieve
>>>>>> the
>>>>>> same?
>>>>>> >>>
>>>>>> >>> Basically I need to turn bunch of numbers x1, ... , x6 into a
>>>>>> matrix like this:
>>>>>> >>>
>>>>>> >>> | e^x1 0 0 |
>>>>>> >>> | x2 e^x3 0 |
>>>>>> >>> | x4 x5 e^x4 |
>>>>>> >>>
>>>>>> >>> i.e. diagonal must be filled with e^xi and lower triangle must be
>>>>>> filled with just xi. Order of x-s is not particularly important, as
>>>>>> these
>>>>>> are learned weights anyway.
>>>>>> >>>
>>>>>> >>> Thanks!
>>>>>> >>> Tambet
>>>>>> >>>
>>>>>> >>> --
>>>>>> >>>
>>>>>> >>> ---
>>>>>> >>> You received this message because you are subscribed to the
>>>>>> Google Groups "theano-users" group.
>>>>>> >>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> >>>
>>>>>> >>> For more options, visit https://groups.google.com/d/optout.
>>>>>> >>
>>>>>> >>
>>>>>> > --
>>>>>> >
>>>>>> > ---
>>>>>> > You received this message because you are subscribed to the Google
>>>>>> Groups "theano-users" group.
>>>>>> > To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> > For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>> --
>>>>>
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "theano-users" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "theano-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
