Great, thanks Stephen.  This is definitely an area that can use some
attention, and I'm glad to see that you are looking into alternatives.

The metadata associated with a recording includes "subject", so it's
certainly possible to load word lists specific to a given subject, and check
whether a string obtained OCR or speech to text is a member of the generic
language and the subject-specific union.  The dictionary API would need to
change, but I don't see a problem with that.

Good point re: loading the word lists from filesystem instead of the
classloader.  The dictionary impl can be reloaded at runtime, though, so it
shouldn't be too much of a hassle to reload new word lists.

Any takers for working on this?  Should I create a branch?  Should we set up
something on github? ;)

Thanks,
Josh

On Mon, Mar 7, 2011 at 9:19 AM, Stephen Marquard <[email protected]
> wrote:

> Hi,
>
> I've recently been looking at dictionaries and language modelling as
> part of preparatory work to integrating Sphinx for speech recognition.
> The issues are similar to OCR recognition, and in fact related because
> some of the approaches to language modelling for speech recognition
> start by using keywords derived from accompanying slides.
>
> One challenge is that many university lectures can be expected to
> contain specialized terms and vocabulary, which a restricted dictionary
> will not include. OTOH a very large dictionary, especially from a corpus
> such as Wikipedia, runs the risk of including many incorrect spellings
> and non-words. However, we can expect that the specialized terms are
> probably those most likely to be searched for, rather than common
> English words.
>
> So far I've looked at a Wikipedia-derived dictionary, a dictionary from
> the Google n-gram corpus at http://ngrams.googlelabs.com/datasets
> (derived from scanned books in Google Books published up to 2000) and
> one or two others.
>
> Using a manual transcription of a lecture on chemical pathology of the
> liver as a reference text, it seems one needs a dictionary of around
> 250K to 500K terms to cover almost all generic terms, and larger than
> that to cover almost all technical terms. These are "naive" dictionaries
> that include all variants rather than stem words, etc.
>
> I'm not really at the "recommend solution" stage of investigations yet,
> but for now would say that it would probably be useful to allow the
> dictionary to be loaded from the filesystem rather than classpath so
> people can experiment with different dictionaries.
>
> Also we should support multiple languages - I'm not sure if recordings
> have that as metadata yet?
>
> I'm happy to share my derived Wikipedia and Google N-gram English
> dictionaries if anyone else would like to experiment. Wikipedia is GFDL
> / CC-BY-SA and the Google n-gram data is CC-BY.
>
> Regards
> Stephen
>
>
> --
> Stephen Marquard, Learning Technologies Co-ordinator
> Centre for Educational Technology, University of Cape Town
> http://www.cet.uct.ac.za
> Email / IM (Jabber/XMPP): [email protected]
> Phone: +27-21-650-5037 Cell: +27-83-500-5290
>
>
> >>> Josh Holtzman <[email protected]> 3/7/2011 9:27 AM >>>
> I've implemented a simple, in-memory dictionary service in order to
> try
> improving the quality of the text segments as displayed in the player.
> The
> idea is to create a smaller, "quality over quantity" dictionary than
> what we
> get from the Wikipedia extract.  See MH-7077 for more details and the
> attached diff.  This implementation is currently English only, but can
> certainly be extended to support multiple languages.  If anyone is
> interested in working on this, please let me know, and I'll create a
> separate module in svn.
>
> Thanks,
> Josh
>
>
>
>
>
> ###
> UNIVERSITY OF CAPE TOWN
>
> This e-mail is subject to the UCT ICT policies and e-mail disclaimer
> published on our website at
> http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from
> +27 21 650 9111. This e-mail is intended only for the person(s) to whom
> it is addressed. If the e-mail has reached you in error, please notify
> the author. If you are not the intended recipient of the e-mail you may
> not use, disclose, copy, redirect or print the content. If this e-mail
> is not related to the business of UCT it is sent by the sender in the
> sender's individual capacity.
>
> ###
>
> _______________________________________________
> Matterhorn mailing list
> [email protected]
> http://lists.opencastproject.org/mailman/listinfo/matterhorn
>
>
> To unsubscribe please email
> [email protected]
> _______________________________________________
>
_______________________________________________
Matterhorn mailing list
[email protected]
http://lists.opencastproject.org/mailman/listinfo/matterhorn


To unsubscribe please email
[email protected]
_______________________________________________

Reply via email to