[
https://issues.apache.org/jira/browse/MAHOUT-738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated MAHOUT-738:
-----------------------------
Attachment: MAHOUT-739.patch
> Collocation driver has long being statically cast to an int
> -----------------------------------------------------------
>
> Key: MAHOUT-738
> URL: https://issues.apache.org/jira/browse/MAHOUT-738
> Project: Mahout
> Issue Type: Bug
> Components: Math
> Affects Versions: 0.5
> Reporter: peter andrews
> Assignee: Sean Owen
> Priority: Minor
> Fix For: 0.6
>
> Attachments: MAHOUT-739.patch
>
>
> org.apache.mahout.vectorizer.collocations.llr.LLRReducer, which is part of
> the collocation driver, statically casts a long to an int.
> private long ngramTotal;
> ...
> int k11 = ngram.getFrequency(); /* a&b */
> int k12 = gramFreq[0] - ngram.getFrequency(); /* a&!b */
> int k21 = gramFreq[1] - ngram.getFrequency(); /* !b&a */
> int k22 = (int) (ngramTotal - (gramFreq[0] + gramFreq[1] -
> ngram.getFrequency())); /* !a&!b */
> These numbers are then fed into
> org.apache.mahout.math.stats.LogLikelihood
> specifically the function below.
> public static double logLikelihoodRatio(int k11, int k12, int k21, int k22) {
> // note that we have counts here, not probabilities, and that the entropy
> is not normalized.
> double rowEntropy = entropy(k11, k12) + entropy(k21, k22);
> double columnEntropy = entropy(k11, k21) + entropy(k12, k22);
> double matrixEntropy = entropy(k11, k12, k21, k22);
> if (rowEntropy + columnEntropy > matrixEntropy) {
> // round off error
> return 0.0;
> }
> return 2.0 * (matrixEntropy - rowEntropy - columnEntropy);
> }
> In short if the long ngramTotal is larger than Integer.MAX_VALUE (which will
> happen in large datasets), then the driver will either crash or in the case
> that it casts to a negative int, will continue as usual but produce no output
> due to error checking.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
