On Mon, Apr 4, 2011 at 12:54 PM, Andrei Zene <[email protected]> wrote: > Hello everybody! > > My name is Andrei Zene, I am a student in first year at > Technical University of Cluj-Napoca. I heard about Google > Summer of Code at a programming contest last autumn and since > then I wanted to apply. > > The most interesting subject for me would be creating the module > to restore diacritics and accents to input text. I say this > because last year I made a spell-checker for romanian language > for my school project so I am somehow familiar with the subject. > (The spell-checker was an add-in for microsoft office written in > C++. If you think it might be useful to post the spell-checker I > could do that. The Documentation is written in romanian language > but I could translate parts of it if necessary.) > > I do not know PERL right now but I have downloaded the charlifter > source code and I think it looks really nice and i would like to > learn it so the porting to C++ should hopefully be not too complicated. > > About optimizing smoothing of the statistical models on a > language-by-language basis, could you explain a little bit more > what would that suppose? >
Hi Andrei, Regarding the comment about smoothing... I've been asked about this by a couple of people offlist and I should say that while I think it's an important and interesting detail, it's not central to the aim of the project. That said... When you do statistical machine learning, you learn from training data of one kind or another (in this case, lots of plain text with the correct diacritics in place). The problem is that no corpus is so big that you'll see every possibility in training, so you need to assign some non-zero probability to the possibilities not seen in training. The best way to do this varies from system to system and in this case from language to language. The problem is that in training the 100+ language models I didn't take the time to "tune" the smoothing to give the best performance for each language. So there are some percentage points of performance there for the taking, I think. Some basics: http://en.wikipedia.org/wiki/Additive_smoothing http://en.wikipedia.org/wiki/Good%E2%80%93Turing_frequency_estimation Your proposal should include code for evaluation of your system in any case (the charlifter perl code does 10-fold cross validation on the training data). So what I had in mind won't involve much extra code - just taking the time to tweak the smoothing parameters and evaluate until you achieve the best performance, language by language. Kevin ------------------------------------------------------------------------------ Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev _______________________________________________ Apertium-stuff mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/apertium-stuff
