Hello Jian,
NPLM reports the log-likelihood of the whole training set, and the
number is plausible.
assuming you have a minibatch size of 1000, your training set perplexity
is exp(1.38122e+08/52853/1000)=13.64
you probably want to measure perplexity on a held-out development set
though, with softmax normalisation instead of NCE.
best wishes,
Rico
On 21.07.2015 13:15, jian zhang wrote:
Hi all,
I am running experiments on Bilingual Neural LM.
For extract_training.py, I set
--prune-target-vocab 10000 --prune-source-vocab 10000 --target-context
5 --source-context 4
For train_nplm.py, I set
--ngram-size 14 --output-embedding 512 --input-embedding 192 --hidden
512 --e 5
I use 2 million parallel sentence pairs for training. The
implementation is from https://github.com/rsennrich/nplm
From the training log generated by train_nplm.py at the first 2
iterations, I have
Number of training minibatches: 52853
Epoch 1
Current learning rate: 1
Training minibatches: 10000...20000...30000...40000...50000...done.
Training NCE log-likelihood: -1.64277e+08
Writing model
Epoch 2
Current learning rate: 1
Training minibatches: 10000...20000...30000...40000...50000...done.
Training NCE log-likelihood: -1.38122e+08
Writing model
The NCE log-likelihood number is suspicious. It is very low. Did I set
any parameters wrong?
Regards,
Jian
--
Jian Zhang
Centre for Next Generation Localisation (CNGL)
<http://www.cngl.ie/index.html>
Dublin City University <http://www.dcu.ie/>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
