Hi Marcin There was a project at MTM2012 for this, but I have not seen any outputs from it http://www.statmt.org/mtm12/index.php%3Fn=Projects.NewDevelopmentFuncionalityForTheAsiyaSuiteParameterOptimizationWithMert I am not aware of anyone working on new metrics for pro and kbmira.
In principle I don't think it would be hard to implement. The current implementations of pro and kbmira make use of the sufficient statistics in the same way that mert does. The main difference is that they require evaluations of single sentences, as opposed to mert which can optimise a corpus metric. kbmira uses Chiang's technique (from his 2008 mira paper) to approximate corpus bleu, but pro just optimises sentence bleu. However it could (and perhaps should) also employ Chiang's technique. Both pro and kbmira use methods from BleuScorer to score the sentences -- smoothedSentenceBleu() and sentenceLevelBackgroundBleu() respectively. cheers - Barry On 09/02/14 09:19, Marcin Junczys-Dowmunt wrote: > Hi list, > It seems that currently for both, pro and kbmira, optimization of BLEU > is hardwired into the code. I managed to add my custom metric to mert, > but would like to experiment with it and sparse features, too. > > I see custom metrics is on a TODO list in the mert folder, is someone > working on custom metrics for sparse features? > Are pro and/or kbmira in principle compatible with this "sufficient > statistics per sentence" approach as it is done for mert? Any pointers > how I could best attack this? > > Best, > Marcin > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
