Hi Ted, I need to tokenize Hindi, an Indian language. I learnt from Robin earlier that "Classifier supports non english tokens(its assumes string is Utf8 encoded)", Does that mean that the Classifier would just tokenize based on unicode encoding, so that we do not need to worry about the language? Or, we do need to make some configurations?
I do not have a knowledge of factors that makes a language harder to tokenize. But, I have learnt from earlier conversations in this mailing list, that languages in which a word is represented as multi-worded (sequence of words), are hard to tokenize. In that sense, I can assume that words in Hindi would be single words. Thanks Bhaskar Ghosh Hyderabad, India http://www.google.com/profiles/bjgindia "Ignorance is Bliss... Knowledge never brings Peace!!!" ________________________________ From: Ted Dunning <[email protected]> To: [email protected] Sent: Sun, 3 October, 2010 12:53:37 AM Subject: Re: How to get multi-language support for training/classifying text into classes through Mahout? You will need to make sure that the tokenization is done reasonable. There is an example program for a sequential classifier in org.apache.mahout.classifiers.sgd.TrainNewsGroups It assumes data in the 20 news groups format and uses a Lucene tokenizer. The NaiveBayes code also uses a Lucene tokenizer that you can specify on the command line. Can you say which languages? Are they easy to tokenize (like French)? Or medium (like German/Turkish)? Or hard (like Chinese/Japanese)? Can you say how much data? On Sat, Oct 2, 2010 at 8:46 AM, Bhaskar Ghosh <[email protected]> wrote: > Dear All, > > I have a requirement where I need to classify text in a non-English > language. I > have heard that Mahout supports multi-language. Can anyone please tell me > how do > I achieve this? Some documents/links where I can get some examples on this, > would be really really helpful. > Regards > Bhaskar Ghosh > Hyderabad, India > > http://www.google.com/profiles/bjgindia > > "Ignorance is Bliss... Knowledge never brings Peace!!!" > > >
