Hello, I was following the instructions here http://www.statmt.org/wmt07/baseline.html , but it is not clearly to me exactly which part of this Europarl data is used for this example.
Nonetheless I downloaded some from 2006 and tried it, but I got many errors about malformed utf8: [EMAIL PROTECTED] moses]$ scripts/tokenizer.perl -l en < moses/wmt07/europarl.fr-en.en > moses/working-dir/corpus/europarl.tok.en Tokenizer Version 1.0 Language: en utf8 "\xE1" does not map to Unicode at scripts/tokenizer.perl line 38, <STDIN> line 88. Malformed UTF-8 character (fatal) at scripts/tokenizer.perl line 53, <STDIN> line 88. I am really surprised the supplied data is not clean! Does everybody clean up the data themselves by hand, or is there a script somewhere? Anyway... as I want to trial Moses with Austronesian languages the Europarl data is not so helpful. I started using a pair of files I prepared myself which have around 1400 parallel sentences/phrases, one per line. Some are very short, like single words. I was doing OK, and followed all the other commands successfully, until I got to the ngram-count command: [EMAIL PROTECTED] moses]$ ~/srilm/bin/i686/ngram-count -order 5 -interpolate -kndiscount -text working-dir/lm/raw.lowercased -lm working-dir/lm/raw.lm one of modified KneserNey discounts is negative error in discount estimator for order 5 The use of clean-corpus-n.perl with parameters 1 & 40 would suggest to me that very short sentences are OK, so any suggestions about what is causing this error or how to fix it? thanks, Brianna _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
