Tom, that's a good point. Henry, you can also check your phrase table with queryPhraseTable to track back the entry that may cause the issue.
-----Ursprüngliche Nachricht----- Von: [email protected] [mailto:[email protected]] Im Auftrag von Tom Hoar Gesendet: 31 July 2012 16:58 An: [email protected] Betreff: Re: [Moses-support] Placeholder drift John, this is true if there were three tokens, but {}Processor{} has no spaces. Assuming that the target language should be {}processeur{} without spaces in both the parallel and LM data, the tables and the language model will treat it as one token and not break break it up. Henry, I suspect your corpus preparation inserts spaces between to create {} Processor {} (3 tokens). John's description is much more viable if this is the case. One oddity is the output {}{} token because it's one token, not two. Moses won't remove the space to splice the two. It would seem your target data contains this as a token from somewhere in the tables or LM. I suggest you double-check your tokenization and other preparation to ensure source and target are still one token when you start training. Tom On Tue, 31 Jul 2012 10:08:43 -0400, John D Burger <[email protected]> wrote: > Are there any such placeholders in your language modeling data and > your parallel training data? If not, all the models are going to > treat them as unknown words. In the case of the language model, it > doesn't surprise me too much that the placeholders all get pushed > together, as that will produce fewer discontiguous subsequences, which > the language model will prefer. > > - John Burger > MITRE > > On Jul 31, 2012, at 03:05 , Henry Hu wrote: > >> Hi, >> >> I use a model to translate English to French. First, I replaced HTML >> tags such as <a>, <b>, with the placeholder {}, like this: >> >> {}Processor{} >> >> Then decoding. To my confusion, I got the result: >> >> {}{} processeur >> >> instead of {}processeur{}. Why did the placeholder move? How can I >> make it fixed? Thanks for any suggestion. >> >> Henry >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
