Hi Tomas
There's some support in the tokeniser for removing special characters,
but it may not deal with all the unicode control characters. In general,
it's better to remove such characters before you pass the data to Moses
as they can cause problems with the training pipeline. Since a lot of
the text processing is in perl, it really depends on how perl handles them,
cheers - Barry
On 07/03/13 10:53, Tomas Hudik wrote:
Hi,
How is moses treating special Unicode control or white space characters?
e.g. http://www.fileformat.info/info/unicode/char/2028/index.htm
<http://www.fileformat.info/info/unicode/char/2028/index.htm>, or
http://en.wikipedia.org/wiki/Unicode_control_characters
<http://en.wikipedia.org/wiki/Unicode_control_characters>
Is it excluded or they become part of phrase table?
Not sure if this question wouldn't be better put somewhere in boost
library forum ...
Thanks, Tomas
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
