Hi Brianna, the 2006 data is not in unicode, so you will have problems using some of the 2007 scripts. Check the 2007 data.
About the ngram-count error - I am not sure what causes this, I did come across it once before, so I'd be interested what you (or someone else) can find out about that. -phi On 29 Nov 2007 18:34:12 +1100, Brianna Laugher <[EMAIL PROTECTED]> wrote: > Hello, > > I was following the instructions here > http://www.statmt.org/wmt07/baseline.html , but it is not clearly to me > exactly which part of this Europarl data is used for this example. > > Nonetheless I downloaded some from 2006 and tried it, but I got many > errors about malformed utf8: > > [EMAIL PROTECTED] moses]$ scripts/tokenizer.perl -l en < > moses/wmt07/europarl.fr-en.en > moses/working-dir/corpus/europarl.tok.en > Tokenizer Version 1.0 > Language: en > utf8 "\xE1" does not map to Unicode at scripts/tokenizer.perl line 38, > <STDIN> line 88. > Malformed UTF-8 character (fatal) at scripts/tokenizer.perl line 53, > <STDIN> line 88. > > I am really surprised the supplied data is not clean! Does everybody > clean up the data themselves by hand, or is there a script somewhere? > > Anyway... as I want to trial Moses with Austronesian languages the > Europarl data is not so helpful. I started using a pair of files I > prepared myself which have around 1400 parallel sentences/phrases, one > per line. Some are very short, like single words. > > I was doing OK, and followed all the other commands successfully, until > I got to the ngram-count command: > > [EMAIL PROTECTED] moses]$ ~/srilm/bin/i686/ngram-count -order 5 -interpolate > -kndiscount -text working-dir/lm/raw.lowercased -lm > working-dir/lm/raw.lm > one of modified KneserNey discounts is negative > error in discount estimator for order 5 > > > The use of clean-corpus-n.perl with parameters 1 & 40 would suggest to > me that very short sentences are OK, so any suggestions about what is > causing this error or how to fix it? > > thanks, > Brianna > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
