We are pleased to announce a new NIST effort designed to *evaluate
metrics *that are employed for the evaluation of MT technology.
See: http://www.nist.gov/speech/tests/metricsmatr/ for current
plans and the evaluation specification document.
The Basics:
(1) NIST will distribute development-data to metric developers
- development-data consists of system translations, references
translations and human assessments of adequacy and preferences.
(2) Metric developers create innovative MT metrics that address the
flaws of current MT metrics
(3) NIST installs the new metric software locally, and processes a large
evaluation-data set
- evaluation-data will be similar to but more expansive than the
development data, and is not to be released.
(4) Reports of correlation studies will be discussed at the AMTA-2008
Workshop: "Metrics MATR: The Metrics for MAchine TRanslation challenge".
For more specific information please visit the link above, and feel free
to address related comments or questions to "[EMAIL PROTECTED]".
Thank You.
Mark Przybocki (for the NIST Open MT Team)
--
// Mark Przybocki - NIST, Phone: 301/975-3347 FAX: 301/670-0939 \\
_______________________________________________
Mt-list mailing list
