The formulation where p(w_i | c_j) is estimated using (eps + count(w_i |
c_j) ) / (eps * N + size(c_j) ) is related to a Bayesian estimator for the
probability with a prior distribution of the probability of each word that
has parameters eps. It behaves very well with zero counts and has long
standing. One of the great virtues this estimate has in a pragmatic sense
is that it does not produce estimates that are either zero or 1.
Using this form avoids the need for adding 1 as you did in your later
formula and will probably give better results.
If you would like to read more on this topic, look up Naive Bayes
classifiers. The wikipedia article on the subject is pretty good except
that they don't detail what they mean by "small sample correction". The
normal way to do that is as I showed above.
http://en.wikipedia.org/wiki/Naive_Bayes_classifier
On Sat, Jun 28, 2008 at 8:45 AM, Maurizio (JIRA) <[EMAIL PROTECTED]> wrote:
>
> [
> https://issues.apache.org/jira/browse/MAHOUT-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609017#action_12609017]
>
> maurizio316 edited comment on MAHOUT-9 at 6/28/08 8:45 AM:
> --------------------------------------------------------
>
> I'm following this direction:
> first, I use unweighted value.
> When I compute P(w_i| C_j), where w_i is a term and C_j is a label, I use
> the follow formula:
> prob(w_i|C_j)=countOcc(w_i,C_j) / size(C_j)
> where
> countOcc function counts # of time that a term occurs in a specific
> category
> and
> size() function counts # of terms within a specific category.
>
> After, I compute bayesian probability in this way:
> argmax_j=log(P(C_j)) + log(prob(w_1,C_j)) + log(prob(w_2,C_j)) +
> log(prob(w_i,C_j))
>
> I'm sure you know better than me, logarithm function prevents underflow
> assignaments. When prob(w_i,C_j) return 0, I throw away this value (in
> short, term that not occurs don't affect final result), this trick prevents
> that logarithm function goes to infinity.
> I tried over 7.000 pre-classified documents and over 100 test document
> and It seems work fine. A further little remark: my document collection is a
> large documents set (laws and legal decisions).
> I'd like to know your opinion.
>
> was (Author: maurizio316):
> I'm following this direction:
> first, I use unweighted value.
> When I compute P(w_i| C_j), where w_i is a term and C_j is a label, I use
> the follow formula:
> prob(w_i|C_j)=countOcc(w_i,C_j) / size(C_j)
> where
> countOcc function counts # of time that a term occurs in a specific
> category
> and
> size() function counts # of terms within a specific category.
>
> After, I compute bayesian probability in this way:
> argmax_j=log(P(C_j)) + log(prob(w_1,C_j)) + log(prob(w_2,C_j)) +
> log(prob(w_i,C_j))
>
> I'm sure you know better than me, logarithm function prevents underflow
> assignaments. When prob(w_i,C_j) return 0, I throw away this value (in
> short, term that not occurs don't affect final result), this trick prevents
> that logarithm function goes to infinity.
> I tried over 7.000 pre-classified documents and over 100 test document
> and It seems work fine.
> I'd like to know your opinion.
>
> > Implement MapReduce BayesianClassifier
> > --------------------------------------
> >
> > Key: MAHOUT-9
> > URL: https://issues.apache.org/jira/browse/MAHOUT-9
> > Project: Mahout
> > Issue Type: New Feature
> > Components: Classification
> > Reporter: Grant Ingersoll
> > Assignee: Grant Ingersoll
> > Priority: Minor
> > Fix For: 0.1
> >
> > Attachments: MAHOUT-9.patch, MAHOUT-9.patch, MAHOUT-9.patch,
> MAHOUT-9.patch, MAHOUT-9.patch
> >
> >
> > Implement a Bayesian classifier using M/R.
> > I have a simple trainer done (not M/R) and will implement the classifier
> soon, then will upgrade it to use Hadoop.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
ted
