Hi Barry, Thanks for your information. I am still not sure about what the 'tokenizing' and the 'detokenizing' is, I mean, what they did and why those handlings are needed. Is the 'tokenizing' something the same with the segmenting?
BTW, I am not familar with Java. Is there any such script wrote by perl/python/C#/C++? And this script just can simply replace the default 'tokenizing' script in the Moses training step, right? Thanks, Wenlong
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
