William, > > Could you post a snippet of the changes. I'm willing to review them > with you. > > James >
Thank you, James. I found the problem while implementing the ChunkerEvaluator. To verify the evaluator I tried to compare the results we get using OpenNLP and the Perl script conlleval available at http://www.cnts.ua.ac.be/conll2000/chunking/output.html. The results where always different if I process more than one string, because the implementation was using FMeasure.updateScores() that was summing divisions. To solve that and have the same results provided by conll I basically stopped using the Mean class. All tests are OK after the change. Index: src/main/java/opennlp/tools/util/eval/FMeasure.java =================================================================== --- src/main/java/opennlp/tools/util/eval/FMeasure.java (revision 1055130) +++ src/main/java/opennlp/tools/util/eval/FMeasure.java (working copy) @@ -18,6 +18,7 @@ package opennlp.tools.util.eval; + /** * The {...@link FMeasure} is an utility class for evaluators * which measure precision, recall and the resulting f-measure. @@ -29,15 +30,13 @@ */ public final class FMeasure { - /** - * The mean of all calculated precision scores. - */ - private Mean precisionScore = new Mean(); - - /** - * The mean of all calculated recall scores. - */ - private Mean recallScore = new Mean(); + /** |selected| = tp + fp */ + private long selected; + + /** |target| = tp + fp */ + private long target; + + private long truePositive; /** * Retrieves the arithmetic mean of the precision scores @@ -46,7 +45,7 @@ * @return the arithmetic mean of all precision scores */ public double getPrecisionScore() { - return precisionScore.mean(); + return selected > 0 ? (double)truePositive / (double)selected : 0; } /** @@ -56,7 +55,7 @@ * @return the arithmetic mean of all recall scores */ public double getRecallScore() { - return recallScore.mean(); + return target > 0 ? (double)truePositive / (double)target : 0; } /** @@ -79,19 +78,16 @@ } public void updateScores(Object references[], Object predictions[]) { - - double precision = FMeasure.precision(references, predictions); - if (!Double.isNaN(precision)) - precisionScore.add(precision, references.length); - - double recall = FMeasure.recall(references, predictions); - if (!Double.isNaN(recall)) - recallScore.add(FMeasure.recall(references, predictions), references.length); + + truePositive += countTruePositives(references, predictions); + selected += predictions.length; + target += references.length; } public void mergeInto(FMeasure measure) { - precisionScore.add(measure.getPrecisionScore(), measure.precisionScore.count()); - recallScore.add(measure.getRecallScore(), measure.recallScore.count()); + this.selected += measure.selected; + this.target += measure.target; + this.truePositive += measure.truePositive; } /**
