A followup by Philip Resnik regarding Peter Turney's question about effect of WSD.
-- Ted Pedersen http://www.d.umn.edu/~tpederse ---------- Forwarded message ---------- Date: Mon, 19 Jul 2004 09:34:12 -0400 (EDT) From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Re: [Senseval-discuss] RE: Re: Senseval-3 Panel on Applications of WSD Hey, you guys, no fair: you're stealing my spot on the panel session for the closing day! :-) But really, that's ok -- indeed, I'm very happy to see a discussion already underway, because gets straight to the heart of *both* panel sessions, and, more generally, to the heart of what SENSEVAL is about. I'll enter that discussion with my own question for the "applications" panelists. If my question gets picked, it would logically be asked before the questions Peter raised. (I'd be delighted to get answers by e-mail, too, if anyone on the list has one to offer.) I have spent quite a lot of time over the last year trying to find an application for which traditional WSD has been demonstrated to make a difference in performance. I'll define "traditional WSD" as the task of mapping from words to labels from a pre-existing sense inventory (or to confidence/probability distributions over such labels), consistent with both SENSEVAL and decades of WSD research. I'll define "make a difference" as either (a) providing a statistically significant improvement on a community-accepted *application* evaluation measure (e.g. precision/recall/F-measure in IR), or (b) being incorporated into a deployed commercial product for a profitable company. Other than one or two very small results in IR, I know of no such application. Do you? (Followup: if not, what do you think "traditional WSD" can tell us about how to obtain such results?) This question is quite pertinent to Peter and Rada's discussion. First, I've been at the NIST MT evaluation workshops, and I'm not convinced that either the testbed or the evaluation measure are necessarily biased against systems incorporating traditional/explicit WSD. Rather, I think the issue is very much the same one found in other NLP application areas: under real-world distributions of test data, the "deeper" phenomena underlying WSD are handled well by shallower statistical approximations. (I'll have more to say about this in the panel.) Second, one could consider adopting Rada's idea of picking evaluation environments where statistical systems are deliberately hampered by a lack of data. (These may or may not be "more realistic", since obtaining parallel text is arguably more economical than building knowledge-based resources.) But it's not clear to me that this would help demonstrate the importance of traditional WSD. The approach has already been taken in the NIST MTeval "small data" track, where parallel text for training data is limited to ~100,000 words, and still nobody has stepped up to challenge the statistical systems. Thanks again to Peter and Rada for getting us started with this really foundational discussion, and to the organizers for what looks to be a really great workshop. Looking forward to seeing you all there. Best, Philip ---------------------------------------------------------------- Philip Resnik, Associate Professor Department of Linguistics and Institute for Advanced Computer Studies 1401 Marie Mount Hall UMIACS phone: (301) 405-6760 University of Maryland Linguistics phone: (301) 405-8903 College Park, MD 20742 USA Fax: (301) 314-2644 / (301) 405-7104 http://umiacs.umd.edu/~resnik E-mail: [EMAIL PROTECTED] _______________________________________________ Senseval-discuss mailing list [EMAIL PROTECTED] http://listserv.hum.gu.se/mailman/listinfo/senseval-discuss ------------------------ Yahoo! Groups Sponsor --------------------~--> Yahoo! Domains - Claim yours for only $14.70 http://us.click.yahoo.com/Z1wmxD/DREIAA/yQLSAA/x3XolB/TM --------------------------------------------------------------------~-> Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/nlpatumd/ <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
