Hi all, I have tried using this evaluator that Matous is talking about. The problem is that there is a difference of 0.5-1 BLEU points difference in the value given by the evaluator script and the regular mteval-v13.plscript. Certainly not a case of precision error!
I am evaluating the tokenized lowercase outputs and reference translations, and the difference was observed for more than 15-20 experiments. So not a case of random error either. Any suggestions about what I could possibly doing wrong? - Prasanth 2011/9/26 Matouš Macháček <[email protected]> > Hi, > > There is a tool for calculating metric score for given candidate and > reference file. You can find this tool in Moses repository under the > mert directory. It uses the same metrics as mert so you can use: > > evaluator --sctype TER --reference ref.file --candidate cand.file > > usage: evaluator [options] --reference ref1[,ref2[,ref3...]] > --candidate cand1[,cand2[,cand3...]] > [--sctype|-s] the scorer type (default BLEU) > [--scconfig|-c] configuration string passed to scorer > This is of the form NAME1:VAL1,NAME2:VAL2 etc > [--reference|-R] comma separated list of reference files > [--candidate|-C] comma separated list of candidate files > [--bootstrap|-b] number of booststraped samples (default 0 - no > bootstraping) > [--rseed|-r] the random seed for bootstraping (defaults to system clock) > [--help|-h] print this message and exit > > Matous Machacek > > 2011/9/26 saeed farzi <[email protected]>: > > Hi all, > > > > Can anybody introduce me a tool for calculating TER, and a tool for > > analysizing ouput of moses. > > -- > > S.Farzi, Ph.D. Student > > Natural Language Processing lab, > > School of Electrical and Computer Eng, > > Tehran University > > Tel: +9821-6111-9719 > > _______________________________________________ > > Moses-support mailing list > > [email protected] > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- "Theories have four stages of acceptance. i) this is worthless nonsense; ii) this is an interesting, but perverse, point of view, iii) this is true, but quite unimportant; iv) I always said so." --- J.B.S. Haldane
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
