Miles, It's hard to say.
On the one hand, I would like to think the paper would have still been accepted. The paper introduces a novel technique which allows standard phrase-based translation to make use of syntax via the language model. As far as evaluation, it's obviously disappointing that the corrected paper fails to show an improvement in translation quality according to BLEU score. On the other hand, BLEU score is not everything, and I would hope that as reviewers we don't judge papers solely on the basis of whether they meet the "does it improve BLEU by at least 1 point" criterion. The paper's evaluation section presents compelling positive perplexity results. The perplexity results demonstrate that, especially for out-of-domain text, the syntactic language model is a better model of language than n-gram LMs, and that even better perplexity results can be realized when the syntactic LM is interpolated with n-gram LMs. I would hope that the novelty of this technique (using an incremental generative parser as a syntactic language model for phrase-based translation), coupled with the extremely positive perplexity results would have together been sufficient for the paper to have been accepted. On the other hand, I certainly acknowledge that ACL is extremely competitive. I have no doubts that the fact that the original submission listed a +1 BLEU increase made it easier for the reviewers to recommend the paper for acceptance. It's also worth looking at the fact that the code and data used in this paper were published alongside the publication. This includes the trained syntactic language model file and the tuned moses configuration file. As a scientist, I want to ensure that our literature reflects the truth, to as great a degree as we are able to. I'd like to think that even if I had not found this error, another researcher looking to replicate the results in the paper would have had little difficulty discovering the problem on their own, aided in no small part by the code, data, and configuration files that accompanied the paper. For me, the fact that there are errors underscores the need for all of us to continue to provide the resources necessary to fully reproduce our experiments, to the greatest extent possible. Cheers, Lane On Mon, Mar 26, 2012 at 4:04 PM, Miles Osborne <[email protected]> wrote: > thanks for pointing this out > > do you think your paper would have been accepted had this error been > spotted in time? > > Miles > > On 26 March 2012 20:52, Lane Schwartz <[email protected]> wrote: >> For those interested in the syntactic language model which I >> implemented as part of my 2011 ACL paper "Incremental Syntactic >> Language Models for Phrase-Based Translation", I want to let you know >> that an erratum to the paper has just been posted to the ACL >> Anthology. A link to and summary of the erratum are below: >> >> http://aclweb.org/anthology-new/P/P11/P11-1063e1.pdf >> >> Schwartz et al. (2011) presented a novel technique for incorporating >> syntactic knowledge into phrase-based machine translation through >> incremental syntactic parsing, and presented empirical results on a >> constrained Urdu-English translation task. The work contained an error >> in the description of the experimental setup, which was discovered >> subsequent to publication. After correcting the error, no improvement >> in BLEU score is seen over the baseline when the syntactic language >> model is used on the constrained Urdu-English translation task. The >> error does not affect the originally reported perplexity results. >> >> Thanks, >> Lane Schwartz >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support > > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, "Time Enough For Love" _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
