On 02/10/13 17:21, John Joseph Morgan wrote: > Hello all, > My understanding is that and end of sentence marker is inserted by the > decoder at some point in the decoding process to give the complete sentence > higher probability than shorter segments of the sentence. > Is this correct?
No. Inserting the eos marker gives the complete sentence lower probability. p(</s> | foo bar .) < 1. It's inserted to model the end of sentence. > If so, can the decoder be configured to not insert the eos marker? > srilm's ngram-count has a -no-eos option, is there a similar option for the > decoder? There is no command line option to disable </s>. > What are the relevant files where this is coded? For phrase-based KenLM, moses/LM/Ken.cpp:255. For phrase-based with other lms, moses/LM/Implementation.cpp near 171. For syntax, see moses/Sentence.cpp near 187 but beware that </s> controls when the glue rule applies. > Thanks, > John > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
