I am using AR-EN GALE data (~5.5 M sentences) for training and tuning on NIST MT06 and testing GALE-DEV07. I also, noticed serious change in the brevity Penalty; from 0.9839 to 0.9137. GALE-DEV07 is two parts; news wire where there is significant improvement by the new MERT implementation but there is degradation on the web part. Till now it is not a big deal. What worried me is the degradation on BLEU and improvement on TER while I am optimizing toward BLEU.
Thanks, Mohamed -----Original Message----- From: Barry Haddow [mailto:[EMAIL PROTECTED] Sent: Wednesday, July 23, 2008 5:34 PM To: [email protected] Cc: Mohamed F. Noamany Subject: Re: [Moses-support] New MERT Hi The new mert is a rewrite of mert to provide a cleaner, more flexible codebase allowing for easier experimentation/extension. It's the same algorithm as the old mert so should give very similar results, however the results won't be exactly the same, and it hasn't been tested as extensively as the old mert so there may still be bugs. What's your train/test setup? Using the fr-en europarl data and testing on the wmt06/07 test sets I got slightly higher bleu scores, but the differences are probably not significant. regards Barry On Wednesday 23 July 2008 22:17:04 Mohamed F. Noamany wrote: > Hi, > Can some one please elaborate more on what has been changed in the MERT > optimization in the latest version (2008-07-08). > > By comparing it to the previous one, I noticed it tends to degrade on BLEU > and improve on TER (comparison on the test/unseen set). I can not > understand that since I am tuning toward BLEU. > Any feedback? > > Thanks, > Mohamed > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
