James, (1) Did you ever look at the model scores? The decoder's job is to find the hypotheses with the highest model score and if your baseline system finds translations with higher model scores than your filtered system then there is no bug in Moses.
(2) You should stop talking about BLEU scores as some kind of evidence that there is a bug in the software. We have an unpublished paper in which we show that using BLEU as the objective function to optimize translations in decoding results in terrible translations, too: http://www2.lingfil.uu.se/SLTC2014/abstracts/sltc2014_submission_21.pdf (3) Tuning is part of the training procedure for log-linear models. There is no point in leaving it out (as many others have told you already). (4) Stop driving on the wrong side of the street ... Jörg On Jun 24, 2015, at 5:21 PM, Read, James C wrote: > May I humbly suggest that we do some market research and see how many > institutions/organisations out there dream about an MT system that out of the > box performs at 37 BLEU points less that merely substituting each phrase for > its most likely translation? I dare say that most users would expect a system > to perform *better* than such a blatantly obvious baseline out of the box. > > So, please, can we stop trying to play the academic high ground here and just > accept that the default behaviour of Moses is much less than desirable? > > James > > > From: Lane Schwartz <dowob...@gmail.com> > Sent: Wednesday, June 24, 2015 5:56 PM > To: Read, James C > Cc: Rico Sennrich; moses-support@mit.edu > Subject: Re: [Moses-support] Major bug found in Moses > > > On Wed, Jun 24, 2015 at 9:05 AM, Read, James C <jcr...@essex.ac.uk> wrote: > As the title of this thread makes clear the purpose of reporting the bug was > not to invite a discussion about conclusions made in my draft paper. Clearly > a community that builds its career around research in SMT is unlikely to > agree with those kinds of conclusions. The purpose was to report the flaw in > the default behaviour of Moses in the hope that we could all agree that > something ought to be done about it. > So far you seem to be the only one who has come even close to acknowledging > that there is a problem with Moses default behaviour. > > James, > > I wasn't talking about the conclusion in your paper. I was talking about the > conclusion in your email: > > If the default behaviour produces BLEU scores considerably lower than merely > selecting the most likely translation of each phrase then evidently there is > something very wrong with the default behaviour. > > Your conclusion, quoted above, is seriously flawed. > > There is not "something very wrong with the default behavior" of Moses. You > have not exposed a bug in Moses. > > What you have exposed is your own lack of understanding of modern statistical > machine translation, and your unwillingness to listen when others take the > time to explain how and why you are mistaken. > > I am happy to help explain things to people who are willing to listen. > However, you have shown yourself to be not only rude but obstinate and > willfully ignorant. I hope that others who find this thread may find it > informative. You appear to have learned nothing from it. > > Until you become willing to listen to others, and until you take a > statistical machine translation class and are willing to pay attention to > what you learn there, I don't see any point in taking the time to explain > things further. As far as I am concerned, this discussion is over. > > Sincerely, > Lane Schwartz > > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
