Hi Jian
You could also try using dropout. Adding something like
--dropout 0.8 --input_dropout 0.9 --null_index 1
to nplm training can help - look at your vocabulary file to see what the
null index should be set to. This works with the Moses version of nplm,
cheers - Barry
On 21/09/15 08:45, Nikolay Bogoychev wrote:
Hey Jian,
I have encountered this problem with nplm myself and couldn't really
find a solution that works every time.
Basically what happens is that there is a token that occurs very
frequently on the same position and it's weights become huge and
eventually not a number which propagates to the rest of the data. This
usually happens with the beginning of sentence token especially if
your source and target size contexts are big. One thing you could do
is to decrease the source and target size context (doesn't always
work). Another thing you could do is to lower the learning rate
(always works, but you might need to set it quite low like 0.25)
The proper solution to this according to Ashish Vasvani who is the
creator of nplm is to use gradient clipping which is commented out in
his code. You should contact him because this is a nplm issue.
Cheers,
Nick
On Sat, Sep 19, 2015 at 8:58 PM, jian zhang <[email protected]
<mailto:[email protected]>> wrote:
Hi all,
I got
Epoch xxxx
Current learning rate: 1
Training minibatches: Validation log-likelihood: -nan
perplexity: nan
during bilingual neural lm training.
I use command:
/home/user/tools/nplm-master-rsennrich/src/trainNeuralNetwork
--train_file work_dir/blm/train.numberized --num_epochs 30
--model_prefix work_dir/blm/train.10k.model.nplm --learning_rate 1
--minibatch_size 1000 --num_noise_samples 100 --num_hidden 2
--input_embedding_dimension 512 --output_embedding_dimension 192
--num_threads 6 --loss_function log --activation_function tanh
--validation_file work_dir/blm/valid.numberized
--validation_minibatch_size 10
where train.numberized and valid.numberized files are splitted
from the file generated by
script ${moses}/scripts/training/bilingual-lm/extract_training.py.
Training/Validation numbers are:
Number of training instances: 4128195
Number of validation instances: 217274
Thanks,
Jian
Jian Zhang
Centre for Next Generation Localisation (CNGL)
<http://www.cngl.ie/index.html>
Dublin City University <http://www.dcu.ie/>
_______________________________________________
Moses-support mailing list
[email protected] <mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
