it's ok to put <s> & </s> into the language model.
However, don't put them into the phrase-based phrase-table. It's ok to
put them into the phrase-table for the chart decoder, which is what the
svn update refers to.
On 09/06/2011 11:53, Tom Hoar wrote:
Hieu,
Your most recent SVN update comment says:
"dont process unknown words for 1st or last place. They're the<s> & </s> and
should only be handled by the glue rules"
We generate language model text corpus files with the<s> and</s> tag on each
line, but we do not add them in 1st/last place. Should they be there? Is it a problem if
they're not there?
<?_task=mail&_id=9934662234df050879222f&_action=compose#>
Thanks,
Tom
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
