Hi,
You should save file with encoding is UTF-8.
Huy.
> From: [EMAIL PROTECTED]
> To: [email protected]
> Date: Thu, 29 Nov 2007 18:34:12 +1100
> Subject: [Moses-support] ngram-count error with -kndiscount option
>
> Hello,
>
> I was following the instructions here
> http://www.statmt.org/wmt07/baseline.html , but it is not clearly to me
> exactly which part of this Europarl data is used for this example.
>
> Nonetheless I downloaded some from 2006 and tried it, but I got many
> errors about malformed utf8:
>
> [EMAIL PROTECTED] moses]$ scripts/tokenizer.perl -l en <
> moses/wmt07/europarl.fr-en.en > moses/working-dir/corpus/europarl.tok.en
> Tokenizer Version 1.0
> Language: en
> utf8 "\xE1" does not map to Unicode at scripts/tokenizer.perl line 38,
> <STDIN> line 88.
> Malformed UTF-8 character (fatal) at scripts/tokenizer.perl line 53,
> <STDIN> line 88.
>
> I am really surprised the supplied data is not clean! Does everybody
> clean up the data themselves by hand, or is there a script somewhere?
>
> Anyway... as I want to trial Moses with Austronesian languages the
> Europarl data is not so helpful. I started using a pair of files I
> prepared myself which have around 1400 parallel sentences/phrases, one
> per line. Some are very short, like single words.
>
> I was doing OK, and followed all the other commands successfully, until
> I got to the ngram-count command:
>
> [EMAIL PROTECTED] moses]$ ~/srilm/bin/i686/ngram-count -order 5 -interpolate
> -kndiscount -text working-dir/lm/raw.lowercased -lm
> working-dir/lm/raw.lm
> one of modified KneserNey discounts is negative
> error in discount estimator for order 5
>
>
> The use of clean-corpus-n.perl with parameters 1 & 40 would suggest to
> me that very short sentences are OK, so any suggestions about what is
> causing this error or how to fix it?
>
> thanks,
> Brianna
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
_________________________________________________________________
Explore the seven wonders of the world
http://search.msn.com/results.aspx?q=7+wonders+world&mkt=en-US&form=QBRE
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
