Very interesting note from Peter Turney regarding the applicability of Word Sense Disambiguation to Machine Translation. This originally went to the Senseval mailing list, so it's ok to circulate.
---------- Forwarded message ---------- Date: Fri, 16 Jul 2004 11:10:04 -0400 From: Peter Turney <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Subject: [Senseval-discuss] RE: Re: Senseval-3 Panel on Applications of WSD Dear Phil: > REQUEST FOR QUESTIONS > SENSEVAL-3 PANEL ON APPLICATIONS OF WSD > 25 July 2004, 16:50-18:00 Question: Machine Translation is often mentioned as a prime application of WSD, but is (explicit) WSD actually helpful for MT? Background: A colleague (George Foster) recently attended the 3rd NIST/DARPA MT Evaluation Workshop near Washington. The following remarks are largely based on his trip report. (I accept all blame for errors and misunderstandings. George should have all credit for a fine trip report.) http://www.nist.gov/speech/tests/mt/index.htm The NIST/DARPA MT Evaluation is one of the main international forums for presentation of MT research. As in previous years, participants competed on Chinese-to-English and/or Arabic-to-English translation, having about 5 days to translate 200 short documents. There are three tracks that determine the resources that can be used to build the competing system: small and large data collections released by the LDC, and "open". In contrast to previous years, test sets were drawn from different genres: newswire (100 docs), editorials (50 docs), and speeches (50 docs). Evaluation of submitted results was performed automatically, using the 4gram case-sensitive BLEU metric calculated from 4 reference translations. There were 17 participants this year, of whom 9 were new. 12 of the systems used statistical techniques. 2 of the systems used explicit WSD -- these 2 were among the worst performing systems. The best system was ISI's. http://www.isi.edu/natural-language/projects/rewrite/ ISI's system is based on statistical machine translation (SMT), using parallel corpora. The SMT systems were among the best systems in the NIST/DARPA MT Evaluation. Most SMT systems perform a kind of implicit WSD. That is, SMT systems take the surrounding context into account when translating a given word. The results of the NIST/DARPA MT Evaluation suggest that implicit WSD (embedded in SMT) is better than explicit WSD (such as Senseval-3 considers). What is the panel's reaction to this? - maybe MT is not a good application for WSD? - maybe WSD is more appropriate for IR? - maybe WSD should be evaluated in the context of MT (e.g., use the NIST/DARPA MT Evaluation framework for evaluating WSD, instead of using the Senseval framework)? - maybe Senseval should switch to application-based evaluation (e.g., WSD for MT, WSD for IR, etc.) and abandon application-free evaluation (i.e., Senseval-1, Senseval-2, Senseval-3)? - maybe the BLEU metric gives misleading results from the point of view of WSD (although it may be fine for MT)? - maybe the MT community and the WSD community should work together to bring explicit WSD into MT systems in a new way, to bring explicit WSD MT up to (and beyond?) the performance of state-of-the-art MT systems? Best wishes, Peter. _______________________________________________ Senseval-discuss mailing list [EMAIL PROTECTED] http://listserv.hum.gu.se/mailman/listinfo/senseval-discuss ------------------------ Yahoo! Groups Sponsor --------------------~--> Yahoo! Domains - Claim yours for only $14.70 http://us.click.yahoo.com/Z1wmxD/DREIAA/yQLSAA/x3XolB/TM --------------------------------------------------------------------~-> Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/nlpatumd/ <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
