Hi, You can use the following command to use tokenizer which exists inside of mosesdecoder folder. Lets your input file (corpora.en) exists inside the "data".folder and the generated tokenized output file name is corpora.tok.en.
~/mosesdecoder/scripts/tokenizer/tokenizer.perl -l en <~/data/corpora.en >~/data/corpora.tok.en On Fri, Mar 23, 2018 at 11:19 PM, Jigyasa Sakhuja <[email protected] > wrote: > hi i want to tokenize my data and i want to use the tokenizer.perl file > but i dont know how to use that tokenizer for tokenization . can you help > me ? > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- Thanks and Regards Debajyoty Banik IIT Patna
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
