You might check tokenizer.perl's new argument: -protected. This option
reads simple regex search patterns from a file and protects the patterns
from tokenization. I've never used it so you'll need to study how it works.
On 01/24/2014 03:58 PM, amir haghighi wrote:
I use the built-in tokenizer in the Moses.
how can I change this tokenizer? should I change the source code?
Regards
Amir
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
