Bridget's thesis defense will be at 3pm on Friday October 30. Here's the
title of her thesis and the associated abstract:
======================================
Extending the Log Likelihood Measure to Improve Collocation Identification
Automatically identifying collocations in a text can be useful for
applications such as machine translation and building lexicons or
knowledge bases. This thesis presents an extension of the Log Likelihood
measure ($G^{2}$) to automatically identify collocations that consist of
more than two words. $G^{2}$ is the ratio between how often an ngram
occurs compared to how often it would be expected to occur given a model.
In the 2-dimensional case, i.e., collocations that consist of only two
words, the only possible model is that of independence. $G^{2}$
calculates the observed count of an ngram and compares it to the count
that would be expected if the words were statistically independent. The
score that $G^{2}$ produces reflects the degree to which the observed
and expected values diverge. Calculating the expected values based on
independence is commonly carried over to the three dimensional case but
as the dimensions grow, so does the number of models in which the words
can be compared to.
Our approach calculates the $G^{2}$ of an ngram for each of the different
possible models and iteratively determines what model `fits'' the ngram
the best. The score the models return allow us to rank the ngrams such
that ngrams that have a high Log Likelihood score are collocations while
ones with low Log Likelihood scores are not.
To calculate the Log Likelihood measure, various co-occurrence and
individual frequency counts of the tokens in the ngram are needed.
Traditionally, the method used for obtaining frequency counts for ngrams
is to count the number of times they appear in a corpus and store them.
However, this becomes very limiting because of memory constraints that
make it infeasible to process large data sets. This problem has lead to
an increasing discussion on the feasibility of using the World Wide Web
as a corpus. We explore the use of document frequencies returned by the
search engines Google and Alta Vista as the term frequencies and
marginal counts of the ngrams in order to calculate ngram statistics.
------------------------ Yahoo! Groups Sponsor --------------------~-->
$9.95 domain names from Yahoo!. Register anything.
http://us.click.yahoo.com/J8kdrA/y20IAA/yQLSAA/x3XolB/TM
--------------------------------------------------------------------~->
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/nlpatumd/
<*> To unsubscribe from this group, send an email to:
[EMAIL PROTECTED]
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
