Below I include a typical moses.ini file. Of course they were kept the same for both runs. The only difference was the phrase table filtering. I did everything in my power to make this the only variable.
James ________________________________________ From: Ondrej Bojar <bo...@ufal.mff.cuni.cz> Sent: Wednesday, June 17, 2015 5:23 PM To: Read, James C; Marcin Junczys-Dowmunt Cc: Moses-support@mit.edu; Arnold, Doug Subject: Re: [Moses-support] Major bug found in Moses Hi, BLEU scores don't mean much, unless you know what the translations look like. Marcin's explanation sounds very plausible. How did you set weights in your experiment? And were they fixed for the two contrastive runs? Cheers, O. On June 17, 2015 4:01:26 PM CEST, "Read, James C" <jcr...@essex.ac.uk> wrote: >Read here for a table of results for 40 language pairs: > > >http://privatewww.essex.ac.uk/~jcread/paper.pdf > > >Would you honestly expect such huge differences in BLEU score? >Honestly!? > > >James > > >________________________________ >From: Read, James C >Sent: Wednesday, June 17, 2015 4:56 PM >To: Marcin Junczys-Dowmunt >Cc: Moses-support@mit.edu; Arnold, Doug >Subject: Re: [Moses-support] Major bug found in Moses > > >You would expect an improvement of 37 BLEU points? > > >James > > >________________________________ >From: Marcin Junczys-Dowmunt <junc...@amu.edu.pl> >Sent: Wednesday, June 17, 2015 4:32 PM >To: Read, James C >Cc: Moses-support@mit.edu; Arnold, Doug >Subject: Re: [Moses-support] Major bug found in Moses > > >Hi James, > >there are many more factors involved than just probability, for >instance word penalties, phrase penalities etc. To be able to validate >your own claim you would need to set weights for all those >non-probabilities to zero. Otherwise there is no hope that moses will >produce anything similar to the most probable translation. And based on >that there is no surprise that there may be different translations. A >pruned phrase table will produce naturally less noise, so I would say >the behaviour you describe is quite exactly what I would expect to >happen. > >Best, > >Marcin > >W dniu 2015-06-17 15:26, Read, James C napisal(a): > >Hi all, > > > >I tried unsuccessfully to publish experiments showing this bug in Moses >behaviour. As a result I have lost interest in attempting to have my >work published. Nonetheless I think you all should be aware of an >anomaly in Moses' behaviour which I have thoroughly exposed and should >be easy enough for you to reproduce. > > > >As I understand it the TM logic of Moses should select the most likely >translations according to the TM. I would therefore expect a run of >Moses with no LM to find sentences which are the most likely or at >least close to the most likely according to the TM. > > > >To test this behaviour I performed two runs of Moses. One with an >unfiltered phrase table the other with a filtered phrase table which >left only the most likely phrase pair for each source language phrase. >The results were truly startling. I observed huge differences in BLEU >score. The filtered phrase tables produced much higher BLEU scores. The >beam size used was the default width of 100. I would not have been >surprised in the differences in BLEU scores where minimal but they were >quite high. > > > >I have been unable to find a logical explanation for this behaviour >other than to conclude that there must be some kind of bug in Moses >which causes a TM only run of Moses to perform poorly in finding the >most likely translations according to the TM when there are less likely >phrase pairs included in the race. > > > >I hope this information will be useful to the Moses community and that >the cause of the behaviour can be found and rectified. > > > >James > > >_______________________________________________ >Moses-support mailing list >Moses-support@mit.edu<mailto:Moses-support@mit.edu> >http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > > > >------------------------------------------------------------------------ > >_______________________________________________ >Moses-support mailing list >Moses-support@mit.edu >http://mailman.mit.edu/mailman/listinfo/moses-support -- Ondrej Bojar (mailto:o...@cuni.cz / bo...@ufal.mff.cuni.cz) http://www.cuni.cz/~obo _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
