+1 on a way to turn off the automatic insertion of the sentence boundary pseudo-tokens, which was implemented a few years ago. I've requested this in the past, but the answer then was the same, that this would be complicated by the rule-based models.
I think this is worth thinking about, though, because in past experiments I found modest but consistent gains in modeling sentence boundaries in the phrase table as well as the language model. The current setup makes this difficult without hacking in strings of multiple pseudo-tokens. - John Burger MITRE On Feb 10, 2013, at 12:43 PM, Kenneth Heafield wrote: > On 02/10/13 17:21, John Joseph Morgan wrote: >> Hello all, >> My understanding is that and end of sentence marker is inserted by the >> decoder at some point in the decoding process to give the complete sentence >> higher probability than shorter segments of the sentence. >> Is this correct? > > No. Inserting the eos marker gives the complete sentence lower > probability. p(</s> | foo bar .) < 1. It's inserted to model the end > of sentence. > >> If so, can the decoder be configured to not insert the eos marker? >> srilm's ngram-count has a -no-eos option, is there a similar option for the >> decoder? > > There is no command line option to disable </s>. > >> What are the relevant files where this is coded? > > For phrase-based KenLM, moses/LM/Ken.cpp:255. For phrase-based with > other lms, moses/LM/Implementation.cpp near 171. For syntax, see > moses/Sentence.cpp near 187 but beware that </s> controls when the glue > rule applies. > >> Thanks, >> John >> >> >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
