Hi, I've recently been looking at dictionaries and language modelling as part of preparatory work to integrating Sphinx for speech recognition. The issues are similar to OCR recognition, and in fact related because some of the approaches to language modelling for speech recognition start by using keywords derived from accompanying slides.
One challenge is that many university lectures can be expected to contain specialized terms and vocabulary, which a restricted dictionary will not include. OTOH a very large dictionary, especially from a corpus such as Wikipedia, runs the risk of including many incorrect spellings and non-words. However, we can expect that the specialized terms are probably those most likely to be searched for, rather than common English words. So far I've looked at a Wikipedia-derived dictionary, a dictionary from the Google n-gram corpus at http://ngrams.googlelabs.com/datasets (derived from scanned books in Google Books published up to 2000) and one or two others. Using a manual transcription of a lecture on chemical pathology of the liver as a reference text, it seems one needs a dictionary of around 250K to 500K terms to cover almost all generic terms, and larger than that to cover almost all technical terms. These are "naive" dictionaries that include all variants rather than stem words, etc. I'm not really at the "recommend solution" stage of investigations yet, but for now would say that it would probably be useful to allow the dictionary to be loaded from the filesystem rather than classpath so people can experiment with different dictionaries. Also we should support multiple languages - I'm not sure if recordings have that as metadata yet? I'm happy to share my derived Wikipedia and Google N-gram English dictionaries if anyone else would like to experiment. Wikipedia is GFDL / CC-BY-SA and the Google n-gram data is CC-BY. Regards Stephen -- Stephen Marquard, Learning Technologies Co-ordinator Centre for Educational Technology, University of Cape Town http://www.cet.uct.ac.za Email / IM (Jabber/XMPP): [email protected] Phone: +27-21-650-5037 Cell: +27-83-500-5290 >>> Josh Holtzman <[email protected]> 3/7/2011 9:27 AM >>> I've implemented a simple, in-memory dictionary service in order to try improving the quality of the text segments as displayed in the player. The idea is to create a smaller, "quality over quantity" dictionary than what we get from the Wikipedia extract. See MH-7077 for more details and the attached diff. This implementation is currently English only, but can certainly be extended to support multiple languages. If anyone is interested in working on this, please let me know, and I'll create a separate module in svn. Thanks, Josh ### UNIVERSITY OF CAPE TOWN This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity. ### _______________________________________________ Matterhorn mailing list [email protected] http://lists.opencastproject.org/mailman/listinfo/matterhorn To unsubscribe please email [email protected] _______________________________________________
