Hi all, Apologies if this has been addressed before, I'm failing to google an answer.
I have a training function where the input varies in size. Some of my training examples are considerably larger than others, so these can result in MemoryErrors. I would like to be able to just ignore these samples, so I'm catching MemoryErrors and then attempting to continue training. My initial issue was that the training function didn't seem to free memory used by intermediate calculations before throwing the MemoryError, so subsequent training examples would have very little memory left to use. Deleting the training function does free up this memory, but then I would have to set up my model/training again using checkpointed parameters (this would be doable but tedious). So instead I tried co-opting theano.function.free() removing the check for allow_gc. This frees the memory... and then seg faults when I try to continue training. So... is there a valid solution to clean up a theano.function after it threw a MemoryError? If not I'll find some workaround where I try to work out the biggest example I can handle for a given network ahead of time. Thanks, David. -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
