The current mert implementation will work with any of the various definitions of BLEU, which differ only in their computation of the brevity penalty. There are three definitions in the current implementation. To switch between these, modify the eff_ref_len variable in scripts/training/cmert-0.5/bleu.py --
"shortest" -- corresponds to NIST BLEU. "closest" -- corresponds to IBM BLEU. "average" -- was used in a previous script, or so I'm told. fwiw, the most recent version of cmert will optimize for TER, provided that you install the TER script yourself (this is not the version currently included in Moses). However, in my limited experience with it the TER script is extremely slow and difficult to use in tuning. I agree that this optimization procedure will not work with very many weights. I recommend using the fewest number of features you can get away with, since it makes the optimization more stable. In another system I have worked with, I found that some of the default features were redundant and could be removed without loss of translation accuracy. The way to identify these is to see if their weights frequently change sign between tuning iterations without substantially impacting the overall score. I don't have enough experience with Moses to know if the same is true there, but I think it's likely. Cheers Adam On 8 May 2008, at 16:44, Miles Osborne wrote: > it should be possible to substitute another scoring function into > MERT, but as far as i'm aware this has not been done (for Moses). > it would be good to see this! > > (ibm-bleu would be another interesting candidate) > > and as for the maximum number of features, your mileage will clearly > vary according to the amount of data etc; as a rule of thumb i've > not seen much using more than say 14 features > > (unlike our large-scale work, which uses millions of features) > > Miles > > 2008/5/8 Jason Katz-Brown <[EMAIL PROTECTED]>: > (I sent this mail once before, but it bounced because one of MIT's > mailman servers filled a disk. So I hope this is not a duplicate.) > > Hi all, > > Has anybody enhanced mert-moses and cmert to allow one to tune for > maximum NIST score, or to maximize some other metric? If not, I am > interested in adding this functionality. > > Also, could somebody give me an idea of the maximum number of feature > weights that is feasible to train with the current mert setup? > > Thanks, > Jason > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
