what language are you tokenizing/detokenizing? can you show an example of a sentence that was incorrectly tokenized/detokenized?
On 28/06/2012 21:55, Henry Hu wrote: > Hi guys, > > In the process of decoding, I usually tokenise material first, then > decoding, at last detokenising. But I found the results of > detokenising were not satisfied, because some blank spaces generated > in tokenising are not got rid of in detokenising. I'm wondering if I > must tokenise material before decoding? > > Thanks in advance. > > Best Regards, > Henry > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
