I think here are two issues. One is how to report your BLEU score, another is how to chose the best weights.
About BLEU score reporting, we run several times, and report average BLEU score But about weights, we may chose the weights which generate the highest BLEU score on dev set, but not the averaged weights. ---------------------------------------------------- Best wishes! Xianhua Li (李贤华) Information Technology Laboratory Fujitsu Research & Development Center Co.,LTD. 13F Tower A, Ocean International Center, No.56 Dong Si Huan Zhong Rd, Chaoyang District, Beijing, China ,100025 E-mail:[email protected] -----邮件原件----- 发件人: [email protected] [mailto:[email protected]] 代表 Jehan Pages 发送时间: 2011年11月23日 16:28 收件人: somayeh bakhshaei 抄送: [email protected] 主题: Re: [Moses-support] tuning problem Hi, On Wed, Nov 23, 2011 at 5:05 PM, somayeh bakhshaei <[email protected]> wrote: > > - should we average all the weights in the various moses.ini generated > during these tunings? Would weights really still make sense doing so? > > ** We do not do this in our lab. We repeat the training phase and then > choose the moses.ini related to the best BLEU tuning. > Yes, it is not a correct job to average the weights, it does not make sense. > Just consider two vector in the space located on two peak of a > function. The average of these two even might be in a valley. Yes. I thought so too. I probably misunderstood what Tom meant by "averaging the final BLEU scores" in this thread. > - should we compare the BLEU values of the various tuning and take > as-is (without modifying it) the moses.ini whose BLEU was the closer > to the average of all the BLEUs? > > ** we choose the best BLEU, hoping we have cached a better optimum > point and use its moses.ini. > Oh ok. So you take the moses.ini with the best BLEU. So I get you have a different method from Tom Hoar and Barry Haddow, who said too in the initial (at least what I think is: http://thread.gmane.org/gmane.comp.nlp.moses.user/5418/focus=5419) topic, who said: "The best plan is to do several runs and take the average bleu." I guess there are several ways to see the problem here. Or maybe I am totally out of my way here and really even more misunderstood what I read here. Thanks anyway. I take good note. :-) Jehan > Best Regards, > > On Wed, Nov 23, 2011 at 8:14 AM, Jehan Pages <[email protected]> wrote: >> >> Hi, >> >> On Tue, Nov 22, 2011 at 10:18 PM, somayeh bakhshaei >> <[email protected]> wrote: >> > Hello, >> > >> > Thanks for all answers. >> > >> > Also thanks Jehan. >> > As you might follow moses emails there is an inconsistency problem >> > about tuning in mert (expressed by Neda) For reducing this problem >> > everyone offered to tune the system repeatedly then choosing the >> > best answer. >> >> Thanks for this explication. Reading Tom Hoar's email, yours and >> researching and finding the original discussion, I am not sure to >> have understood what is the proposed solution: >> >> - should we average all the weights in the various moses.ini >> generated during these tunings? Would weights really still make sense doing >> so? >> >> - should we compare the BLEU values of the various tuning and take >> as-is (without modifying it) the moses.ini whose BLEU was the closer >> to the average of all the BLEUs? >> >> > It is a way of getting rid of local maxima but not exactly catching >> > the global Maxima but instead trapping in another local one :) So I >> > think a better solution is needed! >> >> So if I get it, the logics is that we may get very good BLEU (as from >> what I read, the closer to 1, the better) on some tuning, but they >> are actually local maxima (hence may be in fact terrible against real >> life data). Hence in order to counter this, we prefer to use a tuning >> which made an average BLEU on our data because it would be more >> robust on the long term? >> >> Also, my mathematics are far, but from what I recall, when we want to >> get away from local maxima/minima, one would prefer to use median >> rather than the average (even more on short samples like here), which >> is also very influence by local maxima. Shouldn't it also be the case >> here? >> >> Regards, >> >> Jehan >> >> > >> > On Tue, Nov 22, 2011 at 3:12 PM, Jehan Pages <[email protected]> wrote: >> >> >> >> Hi, >> >> >> >> On Tue, Nov 22, 2011 at 5:57 PM, somayeh bakhshaei >> >> <[email protected]> wrote: >> >> > Hello all, >> >> > >> >> > Salam, >> >> > >> >> > I am using moses in this way: >> >> > >> >> > train, >> >> > for i=1 to 3 >> >> > tune >> >> > end for >> >> >> >> Sorry for not answering your problem (I don't have the solution >> >> though I saw others did answer with a possible resolution). I just >> >> note that you tune 3 times. Do you mean you re-tune using the >> >> exact same data set these 3 times? Does it bring better results to >> >> tune several times like this? >> >> Thanks! >> >> >> >> Jehan >> >> >> >> > decode >> >> > evaluate >> >> > >> >> > in the above loop for something unexpected happens, in large >> >> > execution sometime the weights produced in moses.ini are wrong. >> >> > For example once it produce 3 in the other case produce 4, take >> >> > a look hear: >> >> > >> >> > # translation model weights >> >> > [weight-t] >> >> > 0.0106455 >> >> > 0.036391 >> >> > 0.0453815 >> >> > 0.0716856 >> >> > 0.0271838 >> >> > >> >> > # translation model weights >> >> > [weight-t] >> >> > 0.0705978 >> >> > 0.0652413 >> >> > 0.100475 >> >> > 0.00356951 >> >> > >> >> > in the case in the previous iteration nothing is wrong. >> >> > Did anyone can tell me what is happening here please? >> >> > >> >> > >> >> > >> >> > >> >> > --------------------- >> >> > Best Regards, >> >> > S.Bakhshaei >> >> > >> >> > After All you will come .... >> >> > And will spread light on the dark desolate world! >> >> > O' Kind Father! We will be waiting for your affectionate hands ... >> >> > >> >> > >> >> > _______________________________________________ >> >> > Moses-support mailing list >> >> > [email protected] >> >> > http://mailman.mit.edu/mailman/listinfo/moses-support >> >> > >> >> > >> > >> > >> > >> > -- >> > >> > >> > >> > --------------------- >> > Best Regards, >> > S.Bakhshaei >> > >> > After All you will come .... >> > And will spread light on the dark desolate world! >> > O' Kind Father! We will be waiting for your affectionate hands ... >> > >> > > > > > -- > > > > --------------------- > Best Regards, > S.Bakhshaei > > After All you will come .... > And will spread light on the dark desolate world! > O' Kind Father! We will be waiting for your affectionate hands ... > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
