Sorry, instead of "The results where always different if I process more than one string" read "The results where always different if I process more than one sentence"
On Tue, Jan 4, 2011 at 11:08 PM, [email protected] < [email protected]> wrote: > William, > >> >> Could you post a snippet of the changes. I'm willing to review them >> with you. >> >> James >> > > Thank you, James. > > I found the problem while implementing the ChunkerEvaluator. To verify the > evaluator I tried to compare the results we get using OpenNLP and the Perl > script conlleval available at > http://www.cnts.ua.ac.be/conll2000/chunking/output.html. The results where > always different if I process more than one string, because the > implementation was using FMeasure.updateScores() that was summing divisions. > > To solve that and have the same results provided by conll I basically > stopped using the Mean class. > All tests are OK after the change. > > Index: src/main/java/opennlp/tools/util/eval/FMeasure.java > =================================================================== > --- src/main/java/opennlp/tools/util/eval/FMeasure.java (revision > 1055130) > +++ src/main/java/opennlp/tools/util/eval/FMeasure.java (working copy) > @@ -18,6 +18,7 @@ > package opennlp.tools.util.eval; > > > + > /** > * The {...@link FMeasure} is an utility class for evaluators > * which measure precision, recall and the resulting f-measure. > @@ -29,15 +30,13 @@ > */ > public final class FMeasure { > > - /** > - * The mean of all calculated precision scores. > - */ > - private Mean precisionScore = new Mean(); > - > - /** > - * The mean of all calculated recall scores. > - */ > - private Mean recallScore = new Mean(); > + /** |selected| = tp + fp */ > + private long selected; > + > + /** |target| = tp + fp */ > + private long target; > + > + private long truePositive; > > /** > * Retrieves the arithmetic mean of the precision scores > @@ -46,7 +45,7 @@ > * @return the arithmetic mean of all precision scores > */ > public double getPrecisionScore() { > - return precisionScore.mean(); > + return selected > 0 ? (double)truePositive / (double)selected : 0; > } > > /** > @@ -56,7 +55,7 @@ > * @return the arithmetic mean of all recall scores > */ > public double getRecallScore() { > - return recallScore.mean(); > + return target > 0 ? (double)truePositive / (double)target : 0; > } > > /** > @@ -79,19 +78,16 @@ > } > > public void updateScores(Object references[], Object predictions[]) { > - > - double precision = FMeasure.precision(references, predictions); > - if (!Double.isNaN(precision)) > - precisionScore.add(precision, references.length); > - > - double recall = FMeasure.recall(references, predictions); > - if (!Double.isNaN(recall)) > - recallScore.add(FMeasure.recall(references, predictions), > references.length); > + > + truePositive += countTruePositives(references, predictions); > + selected += predictions.length; > + target += references.length; > } > > public void mergeInto(FMeasure measure) { > - precisionScore.add(measure.getPrecisionScore(), > measure.precisionScore.count()); > - recallScore.add(measure.getRecallScore(), > measure.recallScore.count()); > + this.selected += measure.selected; > + this.target += measure.target; > + this.truePositive += measure.truePositive; > } > > /** > >
