According to the timeline I put in my proposal, I am supposed to start phase 1 today. I want to know which procedures to do to document my work, day by day and week by week. Do I create a page in wiki to save my progress ? Or is there another way ?
Thanks On Fri, Apr 19, 2019 at 9:27 PM Aboelhamd Aly <aboelhamd.abotr...@gmail.com> wrote: > Hi Sevilay. Hi Francis, > > Unfortunately, Sevilay reported that the evaluation results of kaz-tur and > spa-eng pairs were very bad with 30% of the tested sentences were good, > compared to apertium LRLM resolution. > So we discussed what to do next and it is to utilize the breakthrough of > deep learning neural networks in NLP and especially machine translations. > Also we discussed about using different values of n more than 5 in the > already used n-gram language model. And to evaluate the result of > increasing value of n, which could give us some more insights in what to do > next and how to do it. > > Since I have an intro to deep learning subject this term in college, I > waited this past two weeks to be introduced to the application of deep > learning in NLP and MTs. > Now, I have the basics of knowledge in Recurrent Neural Networks (RNNs) > and why to use it instead of the standard network in NLP, beside > understanding the different architectures of it and the math done in the > forward and back propagation. > Also besides knowing how to build a simple language model, and avoiding > the problem of (vanishing gradient) leading to not capturing long > dependencies, by using Gated Recurrent Units (GRus) and Long Short Term > Memory (LSTM) network. > > For next step, we will consider working only on the language model and to > let the max entropy part for later discussions. > So along with trying different n values in the n-gram language model and > evaluate the results, I will try either to use a ready RNNLM or to build a > new one from scratch from what I learnt so far. Honestly I prefer the last > choice because it will increase my experience in applying what I have > learnt. > In last 2 weeks I implemented RNNs with GRUs and LSTM and also implemented > a character based language model as two assignments and they were very fun > to do. So implementing a RNNs word based character LM will not take much > time, though it may not be close to the state-of-the-art model and this is > the disadvantage of it. > > Using NNLM instead of the n-gram LM has these possible advantages : > - Automatically learn such syntactic and semantic features. > - Overcome the curse of dimensionality by generating better > generalizations. > > ---------------------------------------------- > > I tried using n=8 instead of 5 in the n-gram LM, but the scores weren't > that different as Sevilay pointed out in our discussion. > I knew that NNLM is better than statistical one, also that using machine > learning instead of maximum entropy model will give better performance. > *But* the evaluation results were very very disappointing, unexpected and > illogical, so I thought there might be a bug in the code. > And after some search, I found that I did a very very silly *mistake* in > normalizing the LM scores. As the scores are log base 10 of the sentence > probability, then the higher in magnitude has the lower probability, but I > what I did was the inverse of that, and that was the cause of the very bad > results. > > I am fixing this now and then will re-evaluate the results with Sevilay. > > Regards, > Aboelhamd > > > On Sun, Apr 7, 2019 at 6:46 PM Aboelhamd Aly <aboelhamd.abotr...@gmail.com> > wrote: > >> Thanks Sevilay for your feedback, and thanks for the resources. >> >> On Sun, 7 Apr 2019, 18:42 Sevilay Bayatlı <sevilaybaya...@gmail.com >> wrote: >> >>> hi Aboelhamd, >>> >>> Your proposal looks good, I found these resource may be will be benefit. >>> >>> >>> >>> <https://arxiv.org/pdf/1601.00710> >>> Multi-source *neural translation* <https://arxiv.org/abs/1601.00710> >>> https://arxiv.org/abs/1601.00710 >>> >>> >>> <https://arxiv.org/pdf/1708.05943> >>> *Neural machine translation *with extended context >>> <https://arxiv.org/abs/1708.05943> >>> https://arxiv.org/abs/1708.05943 >>> >>> Handling homographs in *neural machine translation* >>> <https://arxiv.org/abs/1708.06510>https://arxiv.org/abs/1708.06510 >>> >>> >>> >>> Sevilay >>> >>> On Sun, Apr 7, 2019 at 7:14 PM Aboelhamd Aly < >>> aboelhamd.abotr...@gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> I got a not solid yet idea as an alternative to yasmet and max entropy >>>> models. >>>> And it's by using neural networks to give us scores for the ambiguous >>>> rules. >>>> But I didn't yet set a formulation for the problem nor the structure of >>>> the inputs, output and even the goal. >>>> As I think there are many formulations that we can adopt. >>>> >>>> For example, the most straightforward structure, is to give the network >>>> all the possible combinations >>>> of a sentence translations and let it choose the best one, or give them >>>> weights. >>>> Hence, make the network learns which combinations to choose for a >>>> specific pair. >>>> >>>> Another example, is instead of building one network per pair, >>>> we build one network per ambiguous pattern as we did with max entropy >>>> models. >>>> So we give to the network the combinations for that pattern, >>>> and let it assign some weights for the ambiguous rules applied to that >>>> pattern. >>>> >>>> And for each structure there are many details and questions to yet >>>> answer. >>>> >>>> So with that said, I decided to look at some papers to see what others >>>> have done before >>>> to tackle some similar problems or the exact problem, and how some of >>>> them used machine learning >>>> or deep learning to solve these problems, and then try build on them. >>>> >>>> Some papers resolution was very specific to the pairs they developed, >>>> thus were not very important to our case. : >>>> 1) Resolving Structural Transfer Ambiguity inChinese-to-Korean Machine >>>> Translation >>>> <https://www.worldscientific.com/doi/10.1142/S0219427903000887>.(2003) >>>> 2) Arabic Machine Translation: A Developmental Perspective >>>> <http://www.ieee.ma/IJICT/IJICT-SI-Bouzoubaa-3.3/2%20-%20paper_farghaly.pdf> >>>> .(2010) >>>> >>>> Some other papers tried not to generate ambiguous rules or to minimize >>>> the ambiguity in transfer rules inference, and didn't provide any methods >>>> to resolve the ambiguity in our case. I thought that they may provide some >>>> help, but I think they are far from our topic : >>>> 1) Learning Transfer Rules for Machine Translation with Limited Data >>>> <http://www.cs.cmu.edu/~kathrin/ThesisSummary/ThesisSummary.pdf>.(2005) >>>> 2) Inferring Shallow-Transfer Machine Translation Rulesfrom Small >>>> Parallel Corpora <https://arxiv.org/pdf/1401.5700.pdf>.(2009) >>>> >>>> Now I am looking into some more recent papers like : >>>> 1) Rule Based Machine Translation Combined with Statistical Post >>>> Editor for Japanese to English Patent Translation >>>> <http://www.mt-archive.info/MTS-2007-Ehara.pdf>.(2007) >>>> 2) Machine translation model using inductive logic programming >>>> <https://scholar.cu.edu.eg/?q=shaalan/files/101.pdf>.(2009) >>>> 3) Machine Learning for Hybrid Machine Translation >>>> <https://www.aclweb.org/anthology/W12-3138.pdf>.(2012) >>>> 4) Study and Comparison of Rule-Based and Statistical Catalan-Spanish >>>> Machine Translation Systems >>>> <https://pdfs.semanticscholar.org/a731/0d0c15b22381c7b372e783d122a5324b005a.pdf?_ga=2.89511443.981790355.1554651923-676013054.1554651923> >>>> .(2012) >>>> 5) Latest trends in hybrid machine translation and its applications >>>> <https://www.sciencedirect.com/science/article/pii/S0885230814001077> >>>> .(2015) >>>> 6) Machine Translation: Phrase-Based, Rule-Based and NeuralApproaches >>>> with Linguistic Evaluation >>>> <http://www.dfki.de/~ansr01/docs/MacketanzEtAl2017_CIT.pdf>.(2017) >>>> 7) A Multitask-Based Neural Machine Translation Model with >>>> Part-of-Speech Tags Integration for Arabic Dialects >>>> <https://www.mdpi.com/2076-3417/8/12/2502/htm>.(2018) >>>> >>>> And I hope they give me some more insights and thoughts. >>>> >>>> -------------- >>>> >>>> - So do you have recommendations to other papers that refer to the same >>>> problem ? >>>> - Also about the proposal, I modified it a little bit and share it >>>> through GSoC website as a draft, >>>> so do you have any last feedback or thoughts about it, or do I just >>>> submit it as a final proposal ? >>>> - Last thing for the coding challenge ( integrating weighted transfer >>>> rules with apertium-transfer ), >>>> I think it's finished, and I didn't get any feedback or response about >>>> it, also the pull-request is not merged yet with master. >>>> >>>> >>>> Thanks, >>>> Aboelhamd >>>> >>>> >>>> On Sat, Apr 6, 2019 at 5:23 AM Aboelhamd Aly < >>>> aboelhamd.abotr...@gmail.com> wrote: >>>> >>>>> Hi Sevilay, hi spectei, >>>>> >>>>> For sentence splitting, I think that we don't need to know neither >>>>> syntax nor sentence boundaries of the language. >>>>> Also I don't see any necessity for applying it in runtime, as in >>>>> runtime we only get the score of each pattern, >>>>> where there is no need for splitting. I also had one thought on using >>>>> beam-search here as I see it has no effect >>>>> and may be I am wrong. We can discuss in it after we close this thread. >>>>> >>>>> We will handle the whole text as one unit and will depend only on the >>>>> captured patterns. >>>>> Knowing that in the chunker terms, successive patterns that don't >>>>> share a transfer rule, are independent. >>>>> So by using the lexical form of the text, we match the words with >>>>> patterns, then match patterns with rules. >>>>> And hence we know which patterns are ambiguous and how much ambiguous >>>>> rules they match. >>>>> >>>>> For example if we have text with the following patterns and >>>>> corresponding rules numbers: >>>>> p1:2 p2:1 p3:6 p4:4 p5:3 p6:5 p7:1 p8:4 p9:4 p10:6 p11:8 >>>>> p12:5 p13:5 p14:1 p15:3 p16:2 >>>>> >>>>> If such text was handled by our old method with generating all the >>>>> combinations possible (multiplication of rules numbers), >>>>> we would have 82944000 possible combinations, which are not practical >>>>> at all to score, and take heavy computations and memory. >>>>> And if it is handled by our new method with applying all ambiguous >>>>> rules of one pattern while fixing the other patterns at LRLM rule >>>>> (addition of rules numbers), we will have just 60 combinations, and >>>>> not all of them different, giving drastically low number of combinations, >>>>> which may be not so representative. >>>>> >>>>> But if we apply the splitting idea , we will have something in the >>>>> middle, that will hopefully avoid the disadvantages of both methods >>>>> and benefit from advantages of both, too. >>>>> Let's proceed from the start of the text to the end of it, while >>>>> maintaining some threshold of say 24000 combinations. >>>>> p1 => 2 ,, p1 p2 => 2 ,, p1 p2 p3 => 12 ,, p1 p2 p3 p4 => >>>>> 48 ,, p1 p2 p3 p4 p5 => 144 ,, >>>>> p1 p2 p3 p4 p5 p6 => 720 ,, p1 p2 p3 p4 p5 p6 p7 => 720 >>>>> p1 p2 p3 p4 p5 p6 p7 p8 => 2880 ,, p1 p2 p3 p4 p5 p6 p7 >>>>> p8 p9 => 11520 >>>>> >>>>> And then we stop here, because taking the next pattern will exceed the >>>>> threshold. >>>>> Hence having our first split, we can now continue our work on it as >>>>> usual. >>>>> But with more -non overwhelming- combinations which would capture more >>>>> semantics. >>>>> After that, we take the next split and so on. >>>>> >>>>> ----------- >>>>> >>>>> I agree with you, that testing the current method with more than one >>>>> pair to know its accuracy is the priority, >>>>> and we currently working on it. >>>>> >>>>> ----------- >>>>> >>>>> For an alternative for yasmet, I agree with spectei. Unfortunately, >>>>> for now I don't have a solid idea to discuss. >>>>> But in the few days, i will try to get one or more ideas to discuss. >>>>> >>>>> >>>>> On Fri, Apr 5, 2019 at 11:23 PM Francis Tyers <fty...@prompsit.com> >>>>> wrote: >>>>> >>>>>> El 2019-04-05 20:57, Sevilay Bayatlı escribió: >>>>>> > On Fri, 5 Apr 2019, 22:41 Francis Tyers, <fty...@prompsit.com> >>>>>> wrote: >>>>>> > >>>>>> >> El 2019-04-05 19:07, Sevilay Bayatlı escribió: >>>>>> >>> Hi Aboelhamd, >>>>>> >>> >>>>>> >>> There is some points in your proposal: >>>>>> >>> >>>>>> >>> First, I do not think "splitting sentence" is a good idea, each >>>>>> >>> language has different syntax, how could you know when you should >>>>>> >>> split the sentence. >>>>>> >> >>>>>> >> Apertium works on the concept of a stream of words, so in the >>>>>> >> runtime >>>>>> >> we can't really rely on robust sentence segmentation. >>>>>> >> >>>>>> >> We can often use it, e.g. for training, but if sentence boundary >>>>>> >> detection >>>>>> >> were to be included, it would need to be trained, as Sevilay hints >>>>>> >> at. >>>>>> >> >>>>>> >> Also, I'm not sure how much we would gain from that. >>>>>> >> >>>>>> >>> Second, "substitute yasmet with other method", I think the result >>>>>> >> will >>>>>> >>> not be more better if you substituted it with statistical method. >>>>>> >>> >>>>>> >> >>>>>> >> Substituting yasmet with a more up to date machine-learning method >>>>>> >> might be a worthwhile thing to do. What suggestions do you have? >>>>>> >> >>>>>> >> I think first we have to trying the exact method with more than 3 >>>>>> >> language pairs and then decide to substitute it or not, because >>>>>> >> what is the point of new method if dont achieve gain, then we can >>>>>> >> compare the results of two methods and choose the best one. What >>>>>> do >>>>>> >> you think? >>>>>> > >>>>>> >>>>>> Yes, testing it with more language pairs is also a priority. >>>>>> >>>>>> Fran >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Apertium-stuff mailing list >>>>>> Apertium-stuff@lists.sourceforge.net >>>>>> https://lists.sourceforge.net/lists/listinfo/apertium-stuff >>>>>> >>>>> _______________________________________________ >>>> Apertium-stuff mailing list >>>> Apertium-stuff@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/apertium-stuff >>>> >>> _______________________________________________ >>> Apertium-stuff mailing list >>> Apertium-stuff@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/apertium-stuff >>> >>
_______________________________________________ Apertium-stuff mailing list Apertium-stuff@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/apertium-stuff
