Hi, this does happen with very large corpora, and Brianna's corpus is pretty large... So there should be many n-grams with low counts.
-phi On Nov 29, 2007 7:33 PM, David Talbot <[EMAIL PROTECTED]> wrote: > Hi Brianna, Phil, > > My guess is that one of the "count of counts" used for computing > discount factors in modified Kneser-Ney is zero. (Modified KN requires > information about the number of n-grams that occurred once, twice and > three times and given the size of the corpus I wouldn't be surprised > if there were no 5-grams that occurred 3 times.) > > A smaller model should work better (or a larger corpus). Try setting > -order 3 (or maybe even 2) in the ngram-count command. > > Cheers, > David > > > Quoting Philipp Koehn <[EMAIL PROTECTED]>: > > > Hi Brianna, > > > > the 2006 data is not in unicode, so you will have problems using > > some of the 2007 scripts. Check the 2007 data. > > > > About the ngram-count error - I am not sure what causes this, > > I did come across it once before, so I'd be interested what you > > (or someone else) can find out about that. > > > > -phi > > > > On 29 Nov 2007 18:34:12 +1100, Brianna Laugher > > <[EMAIL PROTECTED]> wrote: > >> Hello, > >> > >> I was following the instructions here > >> http://www.statmt.org/wmt07/baseline.html , but it is not clearly to me > >> exactly which part of this Europarl data is used for this example. > >> > >> Nonetheless I downloaded some from 2006 and tried it, but I got many > >> errors about malformed utf8: > >> > >> [EMAIL PROTECTED] moses]$ scripts/tokenizer.perl -l en < > >> moses/wmt07/europarl.fr-en.en > moses/working-dir/corpus/europarl.tok.en > >> Tokenizer Version 1.0 > >> Language: en > >> utf8 "\xE1" does not map to Unicode at scripts/tokenizer.perl line 38, > >> <STDIN> line 88. > >> Malformed UTF-8 character (fatal) at scripts/tokenizer.perl line 53, > >> <STDIN> line 88. > >> > >> I am really surprised the supplied data is not clean! Does everybody > >> clean up the data themselves by hand, or is there a script somewhere? > >> > >> Anyway... as I want to trial Moses with Austronesian languages the > >> Europarl data is not so helpful. I started using a pair of files I > >> prepared myself which have around 1400 parallel sentences/phrases, one > >> per line. Some are very short, like single words. > >> > >> I was doing OK, and followed all the other commands successfully, until > >> I got to the ngram-count command: > >> > >> [EMAIL PROTECTED] moses]$ ~/srilm/bin/i686/ngram-count -order 5 > >> -interpolate > >> -kndiscount -text working-dir/lm/raw.lowercased -lm > >> working-dir/lm/raw.lm > >> one of modified KneserNey discounts is negative > >> error in discount estimator for order 5 > >> > >> > >> The use of clean-corpus-n.perl with parameters 1 & 40 would suggest to > >> me that very short sentences are OK, so any suggestions about what is > >> causing this error or how to fix it? > >> > >> thanks, > >> Brianna > >> _______________________________________________ > >> Moses-support mailing list > >> [email protected] > >> http://mailman.mit.edu/mailman/listinfo/moses-support > >> > >> > > _______________________________________________ > > Moses-support mailing list > > [email protected] > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > > > > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
