I'm trying to implement a stateful LSTM in Lasagne, and I'm having trouble dynamically sizing the matrix of initial states.
Basically, I'm defining the initial hidden and cell states as shared variables, each with length equal to number of neurons. It's done this way because the batch size is not known at this time (so we assume a batch size of 1): (1) self.hid_init = theano.shared(np.ones((1,self.num_units), dtype=theano.config.floatX) (2) self.cell_init = theano.shared(np.ones((1,self.num_units), dtype=theano.config.floatX) Now, in the computation graph, I want to resize hid_init and cell_init to batch_size x self.num_units dynamically since the shape info is known. So, I have: (3) hid_init = T.ones((input.shape[0], self.num_units), dtype=theano.config.floatX) * self.hid_init.get_value() (4) cell_init = T.ones((input.shape[0], self.num_units), dtype=theano.config.floatX) * self.cell_init.get_value() Both hid_init and cell_init are provided to the LSTM scan() function as the initial states. I also have a default_update set for self.hid_init and self.cell_init that sets each of these values to the last hidden state/cell state returned from the LSTM scan() function: (5) self.hid_init.default_update = hid_out[:,-1] if not self.only_return_final else hid_out[-1] (6) self.cell_init.default_update = cell_out[-1] I've confirmed that the default_update() portion is working; however, (3) and (4) always returns a matrix of zeros (of the correct size), which indicates that self.hid_init.get_value()in (3) and (4) is perhaps being optimized out of the computation graph. In other words, get_value()always returns the initial row vector of zeros, even though the shared variable is correctly being updated by default_update(). I've spent far too long on this, and would appreciate if anyone could share what might be wrong. Many thanks! -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
