Thanks Barry,
I didn't realize there are perl scripts before Moses binary yesterday.
And after a short discussion with Tom Hoar it seems it would really be better
to filter out all non-text chars (unicode regex: \p{z}, \p{c}). Or, if some
char is important/common in particular dataset re-tag it. Would be difficult
to trace all changes in processing of Unicode by various perl versions, boost
library and potentially some other components involved in moses workflow.
Cheers, t.
From: Barry Haddow [mailto:[email protected]]
Sent: Friday, March 08, 2013 11:30 AM
To: Tomas Hudik
Cc: [email protected]
Subject: Re: [Moses-support] training process and special unicode characters
Hi Tomas
There's some support in the tokeniser for removing special characters, but it
may not deal with all the unicode control characters. In general, it's better
to remove such characters before you pass the data to Moses as they can cause
problems with the training pipeline. Since a lot of the text processing is in
perl, it really depends on how perl handles them,
cheers - Barry
On 07/03/13 10:53, Tomas Hudik wrote:
Hi,
How is moses treating special Unicode control or white space characters?
e.g. http://www.fileformat.info/info/unicode/char/2028/index.htm, or
http://en.wikipedia.org/wiki/Unicode_control_characters
Is it excluded or they become part of phrase table?
Not sure if this question wouldn't be better put somewhere in boost library
forum ...
Thanks, Tomas
_______________________________________________
Moses-support mailing list
[email protected]<mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
