Hi, the scoring tool in MERT relies on the provided tokenization, while mteval-v13.pl does its own tokenization. That may explain the difference.
-phi 2011/10/4 Prasanth K <[email protected]>: > Hi all, > > I have tried using this evaluator that Matous is talking about. > The problem is that there is a difference of 0.5-1 BLEU points difference in > the value given by the evaluator script and the regular mteval-v13.pl > script. > Certainly not a case of precision error! > > I am evaluating the tokenized lowercase outputs and reference translations, > and the difference was observed for more than 15-20 experiments. So not a > case of random error either. > > Any suggestions about what I could possibly doing wrong? > > - Prasanth > > > 2011/9/26 Matouš Macháček <[email protected]> >> >> Hi, >> >> There is a tool for calculating metric score for given candidate and >> reference file. You can find this tool in Moses repository under the >> mert directory. It uses the same metrics as mert so you can use: >> >> evaluator --sctype TER --reference ref.file --candidate cand.file >> >> usage: evaluator [options] --reference ref1[,ref2[,ref3...]] >> --candidate cand1[,cand2[,cand3...]] >> [--sctype|-s] the scorer type (default BLEU) >> [--scconfig|-c] configuration string passed to scorer >> This is of the form NAME1:VAL1,NAME2:VAL2 etc >> [--reference|-R] comma separated list of reference files >> [--candidate|-C] comma separated list of candidate files >> [--bootstrap|-b] number of booststraped samples (default 0 - no >> bootstraping) >> [--rseed|-r] the random seed for bootstraping (defaults to system clock) >> [--help|-h] print this message and exit >> >> Matous Machacek >> >> 2011/9/26 saeed farzi <[email protected]>: >> > Hi all, >> > >> > Can anybody introduce me a tool for calculating TER, and a tool for >> > analysizing ouput of moses. >> > -- >> > S.Farzi, Ph.D. Student >> > Natural Language Processing lab, >> > School of Electrical and Computer Eng, >> > Tehran University >> > Tel: +9821-6111-9719 >> > _______________________________________________ >> > Moses-support mailing list >> > [email protected] >> > http://mailman.mit.edu/mailman/listinfo/moses-support >> > >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support > > > > -- > "Theories have four stages of acceptance. i) this is worthless nonsense; ii) > this is an interesting, but perverse, point of view, iii) this is true, but > quite unimportant; iv) I always said so." > > --- J.B.S. Haldane > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
