The following is a description of our plan of attach for the first stage
of the NSP redesign, that is to organize the measures in an object
oriented hierarchical fashion. The description below is written by Saiyam
Kohli. Your comments and questions are of course most welcome, especially
at this time. We will get started on implementation in the very near
future.
Thanks,
Ted
========================================================================
The Ngram Statistics Package (aka Text-NSP) is being completely rewritten
using object oriented Perl. We will start with the measures for ranking
bigrams and trigrams that are now found in the /Measures directory.
These changes should be transparent to current users of NSP, since
statistic.pl will continue to be used "as is". Internally the Measures
will be organized as a hierarchy of classes, and statistic.pl will serve
as a driver program that calls the methods appropriate for the measures
requested by the user.
The proposed hierarchy for Text-NSP measures is:
Text-NSP::Measure
Text-NSP::Measure::2D
Text-NSP::Measure::2D::fisher
Text-NSP::Measure::2D::fisher::left
Text-NSP::Measure::2D::fisher::right
Text-NSP::Measure::2D::<mi>
Text-NSP::Measure::2D::<mi>::ll
Text-NSP::Measure::2D::<mi>::tmi
Text-NSP::Measure::2D::<mi>::pmi
Text-NSP::Measure::2D::<phi>
Text-NSP::Measure::2D::<phi>::x2
Text-NSP::Measure::2D::<phi>::phi2
Text-NSP::Measure::2D::dice
Text-NSP::Measure::2D::odds
Text-NSP::Measure::2D::tscore
Text-NSP::Measure::3D
Text-NSP::Measure::3D::<mi>::ll
Text-NSP::Measure::3D::<mi>::tmi
Text-NSP::Measure is the base class for all the measures and will
provide basic framework and error checks. Most of the methods in this
class will have to be overridden.
To create a object of any of the measures, statistic.pl will have to pass
the name of that measure to the constructor of this class:
$ll = Text-NSP::Measure->("Text-NSP::Measure::2D::ll");
Since statistic.pl will still be the driver program, it will act as the
abstraction between existing programs that use the ngrams statistics
package and the new implementation. This will also allow users to create
and release their own measures based on the Text-NSP::Measure framework.
Text-NSP::Measure::2D inherits from Text-NSP::Measure and will provide
framework specific for bigram based measures, this class will implement
methods to compute the observed and expected values from the marginal
totals. Error checks that are specific to bi-grams will also be
implemented in this class. Similarly Text-NSP::Measure::3D will provide
framework for trigram measures.
Text-NSP::Measure::2D::fisher will implement methods specific to the
fisher's exact test thus eradicating any duplication of code present in
the current implementation, we will also resolve the errors that have been
reported in the ngram users list regarding Fisher's exact test. (These
errors occur when n22 is not the maximum value in the 2x2 table
representation of the count data).
The computation of the Log-Likelihood measure, Total Mutual Information
and Pointwise Mutual Information is quite similar.
ll = 2* sum[ nij*log(nij/mij)]
tmi = sum[ mij*log(nij/mij)/log2]
pmi = log(n11/m11)
The computations common to these measures will be implemented in
Text-NSP::Measure::2D::<mi> class. Here <mi> is just a placeholder we
have not finalized name for this class.
Similarly, the phi and x2 measures are related to each other, thus the
class <phi> will provide methods for computations and error checks
specific to both these measures.
x2 = npp*phi^2
There was not much common in the rest of the three measures, thus they
will be implemented directly under the Text-NSP::Measure::2D class.
This hierarchy is not final, any suggestions regarding the proposed
changes will be appreciated.
Saiyam Kohli
------------------------ Yahoo! Groups Sponsor --------------------~-->
Give at-risk students the materials they need to succeed at DonorsChoose.org!
http://us.click.yahoo.com/Z1lQfA/LpQLAA/HwKMAA/x3XolB/TM
--------------------------------------------------------------------~->
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/nlpatumd/
<*> To unsubscribe from this group, send an email to:
[EMAIL PROTECTED]
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
