Hey Jian, I have encountered this problem with nplm myself and couldn't really find a solution that works every time.
Basically what happens is that there is a token that occurs very frequently on the same position and it's weights become huge and eventually not a number which propagates to the rest of the data. This usually happens with the beginning of sentence token especially if your source and target size contexts are big. One thing you could do is to decrease the source and target size context (doesn't always work). Another thing you could do is to lower the learning rate (always works, but you might need to set it quite low like 0.25) The proper solution to this according to Ashish Vasvani who is the creator of nplm is to use gradient clipping which is commented out in his code. You should contact him because this is a nplm issue. Cheers, Nick On Sat, Sep 19, 2015 at 8:58 PM, jian zhang <[email protected]> wrote: > Hi all, > > I got > > Epoch xxxx > Current learning rate: 1 > Training minibatches: Validation log-likelihood: -nan > perplexity: nan > > during bilingual neural lm training. > > I use command: > /home/user/tools/nplm-master-rsennrich/src/trainNeuralNetwork --train_file > work_dir/blm/train.numberized --num_epochs 30 --model_prefix > work_dir/blm/train.10k.model.nplm --learning_rate 1 --minibatch_size 1000 > --num_noise_samples 100 --num_hidden 2 --input_embedding_dimension 512 > --output_embedding_dimension 192 --num_threads 6 --loss_function log > --activation_function tanh --validation_file work_dir/blm/valid.numberized > --validation_minibatch_size 10 > > where train.numberized and valid.numberized files are splitted from the > file generated by > script ${moses}/scripts/training/bilingual-lm/extract_training.py. > > Training/Validation numbers are: > Number of training instances: 4128195 > Number of validation instances: 217274 > > > Thanks, > > Jian > > > Jian Zhang > Centre for Next Generation Localisation (CNGL) > <http://www.cngl.ie/index.html> > Dublin City University <http://www.dcu.ie/> > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
