Dear Yuri,
thanks very much for your comments! I will incorporate them with the ones
on the Certification.
At 1:57 PM -0700 10/8/99, Yuri Mordovskoi wrote:
>4. All grammar quality in MT should be based on list of examples that can be
>translated correctly by MT System.
>I can provide list of examples to classified Advanced or Standard system.
>
>5. Difference between MT system SHOULD BASED only on quality of translation
>and dictionaries.
I believe strongly that the Certification should not try to get into the
game of evaluating output quality. That is too sensitive to particular
needs, and hence too easily circumvented (for example, using a set of
test sentences is too easy to create specialized lexicons and grammar
for). Also, quality is not the only feature of interest. So I think
it is better to try to characterize the various kinds of MT system/tool,
and let the user find out whether the quality is good enough for him by
himself.
>"Network ability" is not feature of MT, it is supported software component.
>It can be MT with bad translation quality , but with network function.
>Customer can understand it himself.
Good point.
>6. Also, important point - what happen if Certification agency will assign
>MT as Standard level, but MT Vendor sure for 100% that this is Advanced
>level. After that MT vendor will never come to certification, they simply
>will produce paper where some independent scientist will make statement
>that their MT is Advanced Level. Regular customer will never feel difference
>from
>the professor A from IAMT or the professor B from NOWHERE. It still will be
>all subjective. Only format criteria can resolved this problem and make very
>good background for futire formal evaluation.
In that case, there is a difference between what the Vendor called Advanced
and what the Certification calls Advanced. If the Vendor can get everyone
to agree that his definition of Advanced is more suitable, then the Certi-
fication should be changed. If not, then the Vendor is in the uncomfortable
position of telling everyone his system is great, when perhaps it is not all
that great.
Of course, the criteria must be nonconvtroversial; it is not fair to the
Vendor if the Certifier has some personal opinion. That's why we are
testing the criteria first and then looking for an independent outside
agency to do the real Certifications.
E
>Dear Eduard - if you agree a little bit with my letters, I can prepare
>formal criteria for differenecs between MT systems.
>It can be done based on two idea - complete morphological dictionary (
>50,000 entries) and
>correct translation of basic grammar structures (1,000 typical structures).
>It is only example, but real formal criteria can avoid all subjective
>problems and improve quality of all MT - it will be better for everyone.
>
>Also, I agree with letter of David Mowatt
>
>Yuri Mordovskoi
>CTO
>LanguageForce Inc.
>http://www.languageforce.com
----------------------------------------------------------------------------
Eduard Hovy
email: [EMAIL PROTECTED] USC Information Sciences Institute
tel: 310-822-1511 ext 731 4676 Admiralty Way
fax: 310-823-6714 Marina del Rey, CA 90292-6695
project homepage: http://www.isi.edu/natural-language/nlp-at-isi.html
--
For MT-List info, see http://www.eamt.org/mt-list.html
