Hi Xianhua Way 1 uses no random restarts at all, so each iteration starts where the last one left off, which is why the results are always the same. The other methods both use random restarts (which make them different from Way 1) and they each use fixed seeds (so they are deterministic) but the seeds are different (so they are different from each other).
My guess is that Ways 2 & 3 will give better results as they explore a larger part of the parameter space. However the best method is to follow Clark et al. and run mert several times and average. This advice also applies to most other methods of tuning MT systems, in particular online algorithms where the order of the data may matter. cheers - Barry On Tuesday 01 November 2011 01:42:26 Li Xianhua wrote: > Hi all, > > About the moses mert problem Neda mentioned, I tried three ways to > make MERT deterministic, but their results between them are different. > Way 1: edit mert-moses.pl line 105: "my $_RANDOM-RESTARTS=20", > change 20 to 0 > Way 2: switch to the mert-moses.pl call: --mertargs=" -r $seed " as > Patrik said > Way 3: activate flag "--predictable-seed", as Nicola and Barry said > > I ran totally 9 experiments, 3 for each way with the same corpus and > parameters, and compared the results. the results in the same way are the > same, but the results with the same corpus but different ways are > different. So my questions are: > Are the three ways right to make MERT deterministic? Which is best? > Why are they different? > > ---------------------------------------------------- > Best wishes! > Xianhua Li > Information Technology Laboratory > Fujitsu Research & Development Center Co.,LTD. > 13F Tower A, Ocean International Center, > No.56 Dong Si Huan Zhong Rd, Chaoyang District, Beijing, China ,100025 > E-mail:[email protected] > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- Barry Haddow University of Edinburgh +44 (0) 131 651 3173 -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
