Hi,

I've recently been looking at dictionaries and language modelling as
part of preparatory work to integrating Sphinx for speech recognition.
The issues are similar to OCR recognition, and in fact related because
some of the approaches to language modelling for speech recognition
start by using keywords derived from accompanying slides.

One challenge is that many university lectures can be expected to
contain specialized terms and vocabulary, which a restricted dictionary
will not include. OTOH a very large dictionary, especially from a corpus
such as Wikipedia, runs the risk of including many incorrect spellings
and non-words. However, we can expect that the specialized terms are
probably those most likely to be searched for, rather than common
English words.

So far I've looked at a Wikipedia-derived dictionary, a dictionary from
the Google n-gram corpus at http://ngrams.googlelabs.com/datasets
(derived from scanned books in Google Books published up to 2000) and
one or two others.

Using a manual transcription of a lecture on chemical pathology of the
liver as a reference text, it seems one needs a dictionary of around
250K to 500K terms to cover almost all generic terms, and larger than
that to cover almost all technical terms. These are "naive" dictionaries
that include all variants rather than stem words, etc.

I'm not really at the "recommend solution" stage of investigations yet,
but for now would say that it would probably be useful to allow the
dictionary to be loaded from the filesystem rather than classpath so
people can experiment with different dictionaries.

Also we should support multiple languages - I'm not sure if recordings
have that as metadata yet? 

I'm happy to share my derived Wikipedia and Google N-gram English
dictionaries if anyone else would like to experiment. Wikipedia is GFDL
/ CC-BY-SA and the Google n-gram data is CC-BY.

Regards
Stephen 
 

-- 
Stephen Marquard, Learning Technologies Co-ordinator
Centre for Educational Technology, University of Cape Town
http://www.cet.uct.ac.za
Email / IM (Jabber/XMPP): [email protected]
Phone: +27-21-650-5037 Cell: +27-83-500-5290 


>>> Josh Holtzman <[email protected]> 3/7/2011 9:27 AM >>> 
I've implemented a simple, in-memory dictionary service in order to
try
improving the quality of the text segments as displayed in the player. 
The
idea is to create a smaller, "quality over quantity" dictionary than
what we
get from the Wikipedia extract.  See MH-7077 for more details and the
attached diff.  This implementation is currently English only, but can
certainly be extended to support multiple languages.  If anyone is
interested in working on this, please let me know, and I'll create a
separate module in svn.

Thanks,
Josh



 

###
UNIVERSITY OF CAPE TOWN 

This e-mail is subject to the UCT ICT policies and e-mail disclaimer
published on our website at
http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from
+27 21 650 9111. This e-mail is intended only for the person(s) to whom
it is addressed. If the e-mail has reached you in error, please notify
the author. If you are not the intended recipient of the e-mail you may
not use, disclose, copy, redirect or print the content. If this e-mail
is not related to the business of UCT it is sent by the sender in the
sender's individual capacity.

###
 
_______________________________________________
Matterhorn mailing list
[email protected]
http://lists.opencastproject.org/mailman/listinfo/matterhorn


To unsubscribe please email
[email protected]
_______________________________________________

Reply via email to