Hi Barry,
Thanks for your suggestion. I am doing Japanese-Chinese machine
translation, solving the reordering problem. I want to make sure whether my
method improves the results. Our server is not strong enough and several people
run experiments on it. So I am wondering, can I first run deterministic mert to
make sure the method is effective, then run undeterministic mert several times
and average them to get the results, so that to reduce the server burden?
So the question is like, will the improvement we get while run
deterministic mert still be observed while run undeterministic mert?
----------------------------------------------------
Best wishes!
Xianhua Li
Information Technology Laboratory
Fujitsu Research & Development Center Co.,LTD.
13F Tower A, Ocean International Center,
No.56 Dong Si Huan Zhong Rd, Chaoyang District, Beijing, China ,100025
E-mail:[email protected]
-----邮件原件-----
发件人: Barry Haddow [mailto:[email protected]]
发送时间: 2011年11月1日 16:51
收件人: [email protected]
抄送: Li Xianhua
主题: Re: [Moses-support] mert-moses.pl script
Hi Xianhua
Way 1 uses no random restarts at all, so each iteration starts where the last
one left off, which is why the results are always the same. The other methods
both use random restarts (which make them different from Way 1) and they each
use fixed seeds (so they are deterministic) but the seeds are different (so
they are different from each other).
My guess is that Ways 2 & 3 will give better results as they explore a larger
part of the parameter space. However the best method is to follow Clark et al.
and run mert several times and average. This advice also applies to most other
methods of tuning MT systems, in particular online algorithms where the order
of the data may matter.
cheers - Barry
On Tuesday 01 November 2011 01:42:26 Li Xianhua wrote:
> Hi all,
>
> About the moses mert problem Neda mentioned, I tried three ways to
> make MERT deterministic, but their results between them are different.
> Way 1: edit mert-moses.pl line 105: "my $_RANDOM-RESTARTS=20",
> change 20 to 0
> Way 2: switch to the mert-moses.pl call: --mertargs=" -r $seed " as
> Patrik said
> Way 3: activate flag "--predictable-seed", as Nicola and Barry said
>
> I ran totally 9 experiments, 3 for each way with the same corpus and
> parameters, and compared the results. the results in the same way are
> the same, but the results with the same corpus but different ways are
> different. So my questions are:
> Are the three ways right to make MERT deterministic? Which is best?
> Why are they different?
>
> ----------------------------------------------------
> Best wishes!
> Xianhua Li
> Information Technology Laboratory
> Fujitsu Research & Development Center Co.,LTD.
> 13F Tower A, Ocean International Center,
> No.56 Dong Si Huan Zhong Rd, Chaoyang District, Beijing, China ,100025
> E-mail:[email protected]
>
>
>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
--
Barry Haddow
University of Edinburgh
+44 (0) 131 651 3173
--
The University of Edinburgh is a charitable body, registered in Scotland, with
registration number SC005336.
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
