Hello, Thank s,
mteval-v11b.pl is specified in the step-by-step manual on the moses sit ! Isn't it better to update the site?! ------------------ Best Regards, S.Bakhshaei --- On Sun, 10/17/10, [email protected] <[email protected]> wrote: From: [email protected] <[email protected]> Subject: Moses-support Digest, Vol 48, Issue 28 To: [email protected] Date: Sunday, October 17, 2010, 7:40 PM Send Moses-support mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit http://mailman.mit.edu/mailman/listinfo/moses-support or, via email, send a message with subject or body 'help' to [email protected] You can reach the person managing the list at [email protected] When replying, please edit your Subject line so it is more specific than "Re: Contents of Moses-support digest..." Today's Topics: 1. mteval-v11b (Somayeh Bakhshaei) 2. Re: mteval-v11b (liu chang) 3. Re: mteval-v11b (Miles Osborne) 4. Re: mteval-v11b (Ondrej Bojar) ---------------------------------------------------------------------- Message: 1 Date: Sun, 17 Oct 2010 00:41:24 -0700 (PDT) From: Somayeh Bakhshaei <[email protected]> Subject: [Moses-support] mteval-v11b To: [email protected] Message-ID: <[email protected]> Content-Type: text/plain; charset="us-ascii" Hello, I have some question about mteval-v11b.pl 1) It can not use multi-reference with mteval what is a equivalent tool for this aim? 2) I tried multi-bleu.perl, but the scores reduced ! while we expect to increase while adding more reference sets !! How it is may? 3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they do not always agree ! sometimes mteval and sometimes the other gives better scores. Is there any problem? 4) and at the end, isn't there any better tool with the property of multi-reference? ------------------ Best Regards, S.Bakhshaei -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20101017/34ad975b/attachment-0001.htm ------------------------------ Message: 2 Date: Sun, 17 Oct 2010 16:57:55 +0800 From: liu chang <[email protected]> Subject: Re: [Moses-support] mteval-v11b To: Somayeh Bakhshaei <[email protected]> Cc: [email protected] Message-ID: <[email protected]> Content-Type: text/plain; charset=ISO-8859-1 On Sun, Oct 17, 2010 at 3:41 PM, Somayeh Bakhshaei <[email protected]> wrote: > > Hello, > > I have some question about mteval-v11b.pl > > 1) It can not use multi-reference with mteval what is a equivalent tool for > this aim? > 2) I tried multi-bleu.perl, but the scores reduced ! while we expect to > increase while adding more reference sets !! How it is may? > 3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they > do not always agree ! sometimes mteval and sometimes the other gives better > scores. Is there any problem? > 4) and at the end, isn't there any better tool with the property of > multi-reference? Hi Somayeh, BLEU has defined treatment for multiple references from the very beginning (see the original Papineni et al 2002 paper for details). Any implementation of BLEU that does not support multiple references should be considered defective. Personally I've always used mteval-v13a from http://www.itl.nist.gov/iad/mig/tests/mt/2009/ which has no problem dealing with multiple references at all. All you need to do is to provide the multiple references as multiple doc sections in your reference set: <doc docid="document" sysid="r1"> <seg>...</seg> ... </doc> <doc docid="document" sysid="r2"> ... Disclaimer: The above definitely works for v13a but I'm not specifically familiar with v11b. Cheers, Liu Chang National University of Singapore ------------------------------ Message: 3 Date: Sun, 17 Oct 2010 10:03:23 +0100 From: Miles Osborne <[email protected]> Subject: Re: [Moses-support] mteval-v11b To: liu chang <[email protected]> Cc: [email protected], Somayeh Bakhshaei <[email protected]> Message-ID: <[email protected]> Content-Type: text/plain; charset=ISO-8859-1 note also that NIST changed to IBM BLEU recently which has a different treatment of multiple references. (mteval 13 uses IBM BLEU if i recall) generally the BLEU scores will be a little lower than before, but MERT performance should be more robust Miles On 17 October 2010 09:57, liu chang <[email protected]> wrote: > On Sun, Oct 17, 2010 at 3:41 PM, Somayeh Bakhshaei > <[email protected]> wrote: >> >> Hello, >> >> I have some question about mteval-v11b.pl >> >> 1) It can not use multi-reference with mteval what is a equivalent tool for >> this aim? >> 2) I tried multi-bleu.perl, but the scores reduced ! while we expect to >> increase while adding more reference sets !! How it is may? >> 3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they >> do not always agree ! sometimes mteval and sometimes the other gives better >> scores. Is there any problem? >> 4) and at the end, isn't there any better tool with the property of >> multi-reference? > > Hi Somayeh, > > BLEU has defined treatment for multiple references from the very > beginning (see the original Papineni et al 2002 paper for details). > Any implementation of BLEU that does not support multiple references > should be considered defective. > > Personally I've always used mteval-v13a from > http://www.itl.nist.gov/iad/mig/tests/mt/2009/ which has no problem > dealing with multiple references at all. All you need to do is to > provide the multiple references as multiple doc sections in your > reference set: > > <doc docid="document" sysid="r1"> > ?<seg>...</seg> > ?... > </doc> > <doc docid="document" sysid="r2"> > ... > > Disclaimer: The above definitely works for v13a but I'm not > specifically familiar with v11b. > > Cheers, > Liu Chang > National University of Singapore > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ------------------------------ Message: 4 Date: Sun, 17 Oct 2010 12:29:47 +0200 From: Ondrej Bojar <[email protected]> Subject: Re: [Moses-support] mteval-v11b To: Somayeh Bakhshaei <[email protected]>, [email protected] Message-ID: <[email protected]> Content-Type: text/plain; charset=utf-8 Dear Somayeh, note also that the absolute scores differ heavily based on tokenization (I've seen difference of up to 10 points absolute). mteval-11b does tokenization on its own (possibly tokenizing tokenized input even further), multi-bleu from moses trusts your tokenization. Another difference can come from the definition of "reference length" with multiple translations. Some usethe shortest ref. length, the original paper by Papineni says 'closest' but does not specify *which*! (If the hypothesis is 10 words and two references are 8 and 12 words, which of the two has the closest length?) Implementations differ on this and they even sometimes depend on the *order* of multi references loaded! The main message: never trust the numbers. Compare only BLEU scores you calculated yourself using a fixed tokenization tool and a fixed BLEU implementation. Cheers, O. "Somayeh Bakhshaei" <[email protected]> wrote: >Hello, > >I have some question about mteval-v11b.pl > >1) It can not use multi-reference with mteval what is a equivalent tool for >this aim? >2) I tried multi-bleu.perl, but the scores reduced ! while we expect to >increase while adding more reference sets !! How it is may? >3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they do >not always agree ! sometimes mteval and sometimes the other gives better >scores. Is there any problem? >4) and at the end, isn't there any better tool with the property of >multi-reference? > >------------------ > >Best Regards, > >S.Bakhshaei > > > _______________________________________________ >Moses-support mailing list >[email protected] >http://mailman.mit.edu/mailman/listinfo/moses-support -- Ondrej Bojar http://www.cuni.cz/~obo ------------------------------ _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support End of Moses-support Digest, Vol 48, Issue 28 *********************************************
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
