Awesome, good luck Bridget - I’m sure
it will be great! The abstract sounds very interesting. BTW, just curious, how do
you evaluate collocation identification? Is there a paper on this I could look
at?
Good luck again!
Bano
From: ted pedersen
[mailto:[EMAIL PROTECTED]
Sent: Thursday, October 21, 2004 11:43 PM
To: NLP @ UMD
Subject: [nlpatumd] Bridget's
title/abstract
Bridget's thesis defense will be at 3pm on Friday
October 30. Here's the
title of her thesis and the associated abstract:
======================================
Extending the Log Likelihood Measure to Improve
Collocation Identification
Automatically identifying collocations in a text
can be useful for
applications such as machine translation and
building lexicons or
knowledge bases. This thesis presents an extension
of the Log Likelihood
measure ($G^{2}$) to automatically identify collocations
that consist of
more than two words. $G^{2}$ is the ratio between
how often an ngram
occurs compared to how often it would be expected
to occur given a model.
In the 2-dimensional case, i.e., collocations that
consist of only two
words, the only possible model is that of
independence. $G^{2}$
calculates the observed count of an
ngram and compares it to the count
that would be expected if the words were
statistically independent. The
score that $G^{2}$ produces reflects the
degree to which the observed
and expected values diverge. Calculating the
expected values based on
independence is commonly carried over to the three
dimensional case but
as the dimensions grow, so does the number of
models in which the words
can be compared to.
Our approach calculates the $G^{2}$ of an ngram
for each of the different
possible models and iteratively determines what
model `fits'' the ngram
the best. The score the models return allow
us to rank the ngrams such
that ngrams that have a high Log Likelihood score
are collocations while
ones with low Log Likelihood scores are not.
To calculate the Log Likelihood measure, various
co-occurrence and
individual frequency counts of the tokens in the
ngram are needed.
Traditionally, the method used for obtaining
frequency counts for ngrams
is to count the number of times they appear in a
corpus and store them.
However, this becomes very limiting because
of memory constraints that
make it infeasible to process large data sets.
This problem has lead to
an increasing discussion on the feasibility
of using the World Wide Web
as a corpus. We explore the use of document
frequencies returned by the
search engines Google and Alta Vista as the term
frequencies and
marginal counts of the ngrams in order to
calculate ngram statistics.
| Yahoo! Groups Sponsor |
ADVERTISEMENT
![click here]() | |
![]() |
Yahoo! Groups Links
|
