On Fri, 2007-11-30 at 06:33, David Talbot wrote: > Hi Brianna, Phil, > > My guess is that one of the "count of counts" used for computing > discount factors in modified Kneser-Ney is zero. (Modified KN requires > information about the number of n-grams that occurred once, twice and > three times and given the size of the corpus I wouldn't be surprised > if there were no 5-grams that occurred 3 times.) > > A smaller model should work better (or a larger corpus). Try setting > -order 3 (or maybe even 2) in the ngram-count command.
Thanks David, that fixed it. About the data sets: Training data: wmt07/training/europarl-v3.fr-en.fr wmt07/training/europarl-v3.fr-en.en Format: parallel files, one sentence per line, utf8. (I think I read somewhere they should be at least 1000 sentences) Tuning data: wmt07/dev/dev2006.fr wmt07/dev/dev2006.en Format: same as above?? Size: ?? (compared to training data) Test data: wmt07/devtest/devtest2006.fr wmt07/devtest/devtest2006.en Format: same as above?? Size: ?? Could be whatever size you like I suppose? I think http://www.statmt.org/wmt07/baseline.html these instructions are for translating French -> English only, is that correct? With the "Train Data" command: bin/moses-scripts/scripts-YYYYMMDD-HHMM/training/train-factored-phrase-model.perl -scripts-root-dir bin/moses-scripts/scripts-YYYYMMDD-HHMM -root-dir working-dir -corpus working-dir/corpus/europarl.lowercased -f fr -e en -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm 0:5:working-dir/lm/europarl.lm:0 What is the significance of the flags -f and -e ? Are they related to msd-bidirectional-fe ? Can I just put tg instead of fr and everything else stays the same? (my training files are raw.tg and raw.en) Is it possible to only download the en/fr files rather than the whole Europarl corpus? I only want to use it confirm that I get the expected results and therefore have everything set up correctly. Thanks, Brianna _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
