Hi Brianna, Phil, My guess is that one of the "count of counts" used for computing discount factors in modified Kneser-Ney is zero. (Modified KN requires information about the number of n-grams that occurred once, twice and three times and given the size of the corpus I wouldn't be surprised if there were no 5-grams that occurred 3 times.)
A smaller model should work better (or a larger corpus). Try setting -order 3 (or maybe even 2) in the ngram-count command. Cheers, David Quoting Philipp Koehn <[EMAIL PROTECTED]>: > Hi Brianna, > > the 2006 data is not in unicode, so you will have problems using > some of the 2007 scripts. Check the 2007 data. > > About the ngram-count error - I am not sure what causes this, > I did come across it once before, so I'd be interested what you > (or someone else) can find out about that. > > -phi > > On 29 Nov 2007 18:34:12 +1100, Brianna Laugher > <[EMAIL PROTECTED]> wrote: >> Hello, >> >> I was following the instructions here >> http://www.statmt.org/wmt07/baseline.html , but it is not clearly to me >> exactly which part of this Europarl data is used for this example. >> >> Nonetheless I downloaded some from 2006 and tried it, but I got many >> errors about malformed utf8: >> >> [EMAIL PROTECTED] moses]$ scripts/tokenizer.perl -l en < >> moses/wmt07/europarl.fr-en.en > moses/working-dir/corpus/europarl.tok.en >> Tokenizer Version 1.0 >> Language: en >> utf8 "\xE1" does not map to Unicode at scripts/tokenizer.perl line 38, >> <STDIN> line 88. >> Malformed UTF-8 character (fatal) at scripts/tokenizer.perl line 53, >> <STDIN> line 88. >> >> I am really surprised the supplied data is not clean! Does everybody >> clean up the data themselves by hand, or is there a script somewhere? >> >> Anyway... as I want to trial Moses with Austronesian languages the >> Europarl data is not so helpful. I started using a pair of files I >> prepared myself which have around 1400 parallel sentences/phrases, one >> per line. Some are very short, like single words. >> >> I was doing OK, and followed all the other commands successfully, until >> I got to the ngram-count command: >> >> [EMAIL PROTECTED] moses]$ ~/srilm/bin/i686/ngram-count -order 5 -interpolate >> -kndiscount -text working-dir/lm/raw.lowercased -lm >> working-dir/lm/raw.lm >> one of modified KneserNey discounts is negative >> error in discount estimator for order 5 >> >> >> The use of clean-corpus-n.perl with parameters 1 & 40 would suggest to >> me that very short sentences are OK, so any suggestions about what is >> causing this error or how to fix it? >> >> thanks, >> Brianna >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
