Dears, I have misunderstanding on what tokenization really do
What I think that It makes the translation of text like translated text gives the same output as "translated" text or translated.text or translated text . which ignores any punctuations in the translated text Am I right ? I did the tokenization on my data but this is not happening Note : in the tokenizer script I should feed it with the language and it could not recognize the arabic language (ar) which is my target language Best Regards Ihab Ramadan| Senior Developer| <http://www.saudisoft.com/> Saudisoft - Egypt | Tel +2 02 330 320 37 Ext- 0 | Mob+201007570826 | Fax+20233032036 | Follow us on <http://www.linkedin.com/company/77017?trk=vsrp_companies_res_name&trkInfo=V SRPsearchId%3A1489659901402995947155%2CVSRPtargetId%3A77017%2CVSRPcmpt%3Apri mary> linked | <https://www.facebook.com/pages/Saudisoft-Co-Ltd/289968997768973?ref_type=bo okmark> ZA102637861 | <https://twitter.com/Saudisoft> ZA102637858
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
