Hi, Unfortunately, this is not a case we handle well, and since we do not really test recovery from errors, I would bet many of the Ops do not behave correctly wrt refcounts when encountering a MemoryError, which could explain the segfault.
I think the real solution would be to clean those up (and test...), but I'm afraid it would be too big of a task. On Tue, Aug 23, 2016, David Knowles wrote: > I have a training function where the input varies in size. Some of my > training examples are considerably larger than others, so these can result > in MemoryErrors. I would like to be able to just ignore these samples, so > I'm catching MemoryErrors and then attempting to continue training. > > My initial issue was that the training function didn't seem to free memory > used by intermediate calculations before throwing the MemoryError, so > subsequent training examples would have very little memory left to use. > Deleting the training function does free up this memory, but then I would > have to set up my model/training again using checkpointed parameters (this > would be doable but tedious). So instead I tried co-opting > theano.function.free() removing the check for allow_gc. This frees the > memory... and then seg faults when I try to continue training. > > So... is there a valid solution to clean up a theano.function after it > threw a MemoryError? If not I'll find some workaround where I try to work > out the biggest example I can handle for a given network ahead of time. > > Thanks, > > David. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "theano-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- Pascal -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
