Hi Harold, > Does anyone have any experience of applying BLEU/NIST scores to > non-spacing languages like Japanese, Chinese, Korean? Since they are > word-based metrics, I presume output has to be segmented somehow.
Your question provides my with the perfect opportunity to bring to your attention (and to others on this list) a new MT evaluation metric that my students and I have developed, which we have named METEOR. We just released METEOR as a software package to the general public last week. The website for downloading METEOR is the following: http://www-2.cs.cmu.edu/~alavie/METEOR/ The software includes instructions on how to install and run the code. It's very lightweight, and it follows the exact input file formats that are used by BLEU and NIST. If you know how to run BLEU, you know how to run METEOR. METEOR produces scores that have been demonstrated to have significantly improved correlation with human judgements. We have not tried METEOR on any Asian languages yet, but I have good reasons to believe that when adapted to run on the *character* level, it would work much better than BLEU. This is due to the fact that the metric is based on combined *unigram* precision *and* recall plus an explicit measure capturing how well-ordered the words in the output are with respect to the reference. In the case of the Asian languages that would be done similarly on the character level. I suspect that if we split the (asian lang) input to METEOR to single characters with spaces inserted between each character, METEOR will basically run as is on this, but I will ask one of my students to try this out. If any of you try out METEOR and have any comments or suggestions, please let us know! Best, - *Alon* ----------------------------------------------------------------------------- Dr. Alon Lavie Tel : (+1-412) 268-5655 Language Technologies Institute Fax : (+1-412) 268-6298 Carnegie Mellon University E-mail: [EMAIL PROTECTED] Pittsburgh, PA 15213 USA Homepage: http://www.cs.cmu.edu/~alavie ----------------------------------------------------------------------------- _______________________________________________ Mt-list mailing list
