Well, I am a former student of that pesky Callison-Burch fellow, who keeps muttering about what humans think about the actual translations... ;)
On Mon, Mar 26, 2012 at 5:36 PM, Miles Osborne <[email protected]> wrote: > what a wonderful answer! if only more people were as honest as you are. > > for translation perplexity has a very murky connection with what > people care about, namely our beloved BLEU scores. so good PPL might > not be that interesting to translation people. > > the positive thing however is that papers are not accepted in terms of > scores alone. (in the past i've rejected papers with so-called > improvements when there is no good corresponding argument). and > likewise i've seen papers accepted with no improvements, but with a > good argument. > > Miles > > On 26 March 2012 21:46, Lane Schwartz <[email protected]> wrote: > > Miles, > > > > It's hard to say. > > > > On the one hand, I would like to think the paper would have still been > > accepted. The paper introduces a novel technique which allows standard > > phrase-based translation to make use of syntax via the language model. > > As far as evaluation, it's obviously disappointing that the corrected > > paper fails to show an improvement in translation quality according to > > BLEU score. On the other hand, BLEU score is not everything, and I > > would hope that as reviewers we don't judge papers solely on the basis > > of whether they meet the "does it improve BLEU by at least 1 point" > > criterion. > > > > The paper's evaluation section presents compelling positive perplexity > > results. The perplexity results demonstrate that, especially for > > out-of-domain text, the syntactic language model is a better model of > > language than n-gram LMs, and that even better perplexity results can > > be realized when the syntactic LM is interpolated with n-gram LMs. I > > would hope that the novelty of this technique (using an incremental > > generative parser as a syntactic language model for phrase-based > > translation), coupled with the extremely positive perplexity results > > would have together been sufficient for the paper to have been > > accepted. > > > > On the other hand, I certainly acknowledge that ACL is extremely > > competitive. I have no doubts that the fact that the original > > submission listed a +1 BLEU increase made it easier for the reviewers > > to recommend the paper for acceptance. > > > > It's also worth looking at the fact that the code and data used in > > this paper were published alongside the publication. This includes the > > trained syntactic language model file and the tuned moses > > configuration file. > > > > As a scientist, I want to ensure that our literature reflects the > > truth, to as great a degree as we are able to. I'd like to think that > > even if I had not found this error, another researcher looking to > > replicate the results in the paper would have had little difficulty > > discovering the problem on their own, aided in no small part by the > > code, data, and configuration files that accompanied the paper. > > > > For me, the fact that there are errors underscores the need for all of > > us to continue to provide the resources necessary to fully reproduce > > our experiments, to the greatest extent possible. > > > > Cheers, > > Lane > > > > > > On Mon, Mar 26, 2012 at 4:04 PM, Miles Osborne <[email protected]> > wrote: > >> thanks for pointing this out > >> > >> do you think your paper would have been accepted had this error been > >> spotted in time? > >> > >> Miles > >> > >> On 26 March 2012 20:52, Lane Schwartz <[email protected]> wrote: > >>> For those interested in the syntactic language model which I > >>> implemented as part of my 2011 ACL paper "Incremental Syntactic > >>> Language Models for Phrase-Based Translation", I want to let you know > >>> that an erratum to the paper has just been posted to the ACL > >>> Anthology. A link to and summary of the erratum are below: > >>> > >>> http://aclweb.org/anthology-new/P/P11/P11-1063e1.pdf > >>> > >>> Schwartz et al. (2011) presented a novel technique for incorporating > >>> syntactic knowledge into phrase-based machine translation through > >>> incremental syntactic parsing, and presented empirical results on a > >>> constrained Urdu-English translation task. The work contained an error > >>> in the description of the experimental setup, which was discovered > >>> subsequent to publication. After correcting the error, no improvement > >>> in BLEU score is seen over the baseline when the syntactic language > >>> model is used on the constrained Urdu-English translation task. The > >>> error does not affect the originally reported perplexity results. > >>> > >>> Thanks, > >>> Lane Schwartz > >>> _______________________________________________ > >>> Moses-support mailing list > >>> [email protected] > >>> http://mailman.mit.edu/mailman/listinfo/moses-support > >> > >> > >> > >> -- > >> The University of Edinburgh is a charitable body, registered in > >> Scotland, with registration number SC005336. > > > > > > > > -- > > When a place gets crowded enough to require ID's, social collapse is not > > far away. It is time to go elsewhere. The best thing about space travel > > is that it made it possible to go elsewhere. > > -- R.A. Heinlein, "Time Enough For Love" > > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, "Time Enough For Love"
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
