Hi Barry, OK, thanks for the confirmation, so there is sense to try it. I will see whether I can manage to add my metric (which by itself is not particularly useful to the community) and maybe I will manage to convince pro or kbmira by the way to use the general Scorer classes from mert.
Which one do you prefer for sparse features? How do they cope with optimizer instability compared to mert? Best, Marcin W dniu 09.02.2014 20:53, Barry Haddow pisze: > Hi Marcin > > There was a project at MTM2012 for this, but I have not seen any > outputs from it > http://www.statmt.org/mtm12/index.php%3Fn=Projects.NewDevelopmentFuncionalityForTheAsiyaSuiteParameterOptimizationWithMert > > > I am not aware of anyone working on new metrics for pro and kbmira. > > In principle I don't think it would be hard to implement. The current > implementations of pro and kbmira make use of the sufficient > statistics in the same way that mert does. The main difference is that > they require evaluations of single sentences, as opposed to mert which > can optimise a corpus metric. kbmira uses Chiang's technique (from his > 2008 mira paper) to approximate corpus bleu, but pro just optimises > sentence bleu. However it could (and perhaps should) also employ > Chiang's technique. Both pro and kbmira use methods from BleuScorer to > score the sentences -- smoothedSentenceBleu() and > sentenceLevelBackgroundBleu() respectively. > > cheers - Barry > > On 09/02/14 09:19, Marcin Junczys-Dowmunt wrote: >> Hi list, >> It seems that currently for both, pro and kbmira, optimization of BLEU >> is hardwired into the code. I managed to add my custom metric to mert, >> but would like to experiment with it and sparse features, too. >> >> I see custom metrics is on a TODO list in the mert folder, is someone >> working on custom metrics for sparse features? >> Are pro and/or kbmira in principle compatible with this "sufficient >> statistics per sentence" approach as it is done for mert? Any pointers >> how I could best attack this? >> >> Best, >> Marcin >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
