Hello Marcus,
Regardingthe BaseAugmented class, we should avoid inheritance (or, at least, virtualinheritance) in deference to templates. This is because virtual functions incurruntime overhead. In this case I don't see a compelling reason to introduce aBaseAugmented class, maybe I missed something? We could do:Code:=====class CopyTask{template<typename ModelType>void Evaluate(ModelType& model){// Call model.Train() using the generated copy task data.// Call model.Evaluate}}Usage:======NTM ntm;CopyTask.Evaluate(ntm);What do you think about the design?
This is definitely an excellent idea :) Actually, this shows that I have forgotten most of C++ - here, I forgot about that after using the template keyword, I cal call any functions I like (just like in Python, actually). So, it's about time to refresh C++ - and thanks for your advice on resources, since my C++ experience was biased towards competitive programming, which mostly ignores these language features.
- Can you show me the working example of mlpack::ann::FFN compatible withcurrent upstream C++ API?
Not sure what you mean, can you elaborate more?
Something like this: https://thesundayprogrammer.wordpress.com/2016/01/27/neural-networks-using-mlpack/, but written for the up-to-date mlpack version, rather that the version from last year. Of course, I tried hard to google it, but failed :)
I think it's time to sum up the discussed API: we create namespace mlpack::ann::augmented, and do all the work in it. For instance, we create NTM class with Train() / Evaluate() methods (standard for mlpack). Also, we create classes for evaluating models (for instance, CopyTask).
As an idea: what do you think about creating yet another namespace for such tasks? (for instance, the full class name will be mlpack::ann::augmented::tasks::CopyTask)? Earlier I've offered to inherit them from BaseBenchmark, but now I see it's not the only way. Right now, I'm more inclined to think that not doing inheritance is better because of that argument with the virtual functions. What do you think?
I'm interested in continuing the discussion and getting deeper into that project. By the way, sorry for the delay with the first API idea. The reason is that I'm a freshman student, and for this reason my university studies often get quite stressful. However, right now I'm trying to submit all of my coursework and finish the semester early - with the goal of fully transiting to GSoC work :)
--
Best Regards,
Konstantin.
07.03.2017, 17:54, "Marcus Edel" <[email protected]>:
Hello Konstantin,thanks for getting back.Since we have to make several different augmented RNN models, I think it is agood idea to make a namespace mlpacK::ann::augmented and a class BaseAugmented(will be useful for benchmarking later).Putting different networks under a unified namespace is a good idea. Regardingthe BaseAugmented class, we should avoid inheritance (or, at least, virtualinheritance) in deference to templates. This is because virtual functions incurruntime overhead. In this case I don't see a compelling reason to introduce aBaseAugmented class, maybe I missed something? We could do:Code:=====class CopyTask{template<typename ModelType>void Evaluate(ModelType& model){// Call model.Train() using the generated copy task data.// Call model.Evaluate}}Usage:======NTM ntm;CopyTask.Evaluate(ntm);What do you think about the design?P.S. Some questions that arose while trying to get in grips with mlpack:- What resources can you advice to brush up *advanced/modern* C++? (e.g., templates, && in functions)There are some really nice books that might help to refresh your knowledge:- "Modern C++ Design, Generic Programming and Design Patterns Applied" by Andrei Alexandrescu- "Effective C++" by Scott Meyers- "Effective STL" by Scott MeyersThere are also some references on http://mlpack.org/gsoc.html- Can you show me the working example of mlpack::ann::FFN compatible withcurrent upstream C++ API?Not sure what you mean, can you elaborate more?I hope this is helpful, let us know if you have any more questions.Thanks,MarcusOn 5 Mar 2017, at 10:12, Сидоров Константин <[email protected]> wrote:Hello Marcus!Right now, I'm thinking quite a lot on "Augmented RNNs" project. For example, I'm trying to convert ideas from project description to something more concrete and mlpack-specific. Here is my first shot at C++ API for NTM.Disclaimer: I haven't written on C++ for ~10 months. My last experience of C++ coding is our (Russian) national programming olympiad (April '16), after which I've been coding almost exclusively on Python.Since we have to make several different augmented RNN models, I think it is a good idea to make a namespace mlpacK::ann::augmented and a class BaseAugmented (will be useful for benchmarking later).Inside it, we can define class NTM< OutputLayerType, InitializationRuleType > : BaseAugmented with standard mlpack interface:- template<typename NetworkType, typename Lambda> NTM(NetworkType controller, arma::mat &memory, Lambda similarity)- Predict (arma::mat &predictors, arma::mat &responses)- Train (const arma::mat &predictors, const arma::mat &responses)For benchmarking, we can define class mlpack::ann:augmented::BaseBenchmark with interface:- BaseBenchmark() = 0- Evaluate() = 0 (the method that will accept task parameters as arguments and run the augmented model)As an example, an API of CopyBenchmark : BaseBenchmark:- CopyBenchmark()- Evaluate(BaseAugmented model, int maxLength = 5, int repeats = 1) // repeats is a parameter to convert copy to repeat-copy task.So, that's some kind of API. I would be interested to discuss and analyze it with you.P.S. Some questions that arose while trying to get in grips with mlpack:- What resources can you advice to brush up *advanced/modern* C++? (e.g., templates, && in functions)- Can you show me the working example of mlpack::ann::FFN compatible with current upstream C++ API?28.02.2017, 16:43, "Marcus Edel" <[email protected]>:Hello Konstantin,My name is Konstantin Sidorov, and I am an undergraduate student in AstrakhanState University (Russia). I’m glad to know that mlpack was accepted in GSoC’17– as a side note, congratulations :)thanks and welcome!I’m already fairly familiar with deep learning. For example, recently Iimplemented optimality tightening from “Learning to play in a day”(https://arxiv.org/abs/1611.01606) for the AgentNet (“Deep ReinforcementLearning library for humans”, https://github.com/yandexdataschool/AgentNet).Sounds really interesting, the "Learning to Play in a Day" paper is on myreading list, looks like I should move it up.Of course, on such an early stage I have no detailed plan what (and how) to do –only some ideas. In the beginning, for example, I’m planning to implement NTMsas described in arXiv paper and implement *reusable* benchmarking code (e.g.,copy, repeat copy, n-grams). I would like to discuss this project morethoroughly if possible. In addition, this is my first participation in GSoC. So,excuse me in advance if I’ve done something inappropriate.Implementing the NTM task from the paper, so that they can be used for othermodels as well is a great idea. In fact, you see a lot of other papers that atleast reuse the copy task. There are a bunch of other interesting tasks thatcould be implemented like the MNIST pen stroke classification task recentlyintroduced by Edwin D. de Jong in his "Incremental Sequence Learning" paper. TheStanford Natural Language Inference task proposed by Samuel R. Bowman et al. in"A large annotated corpus for learning natural language inference" can be alsotransformed into a long-term dependency task, that might be interesting.Regarding the project itself, take a look at other models as well, depending onthe model you choose, I think there is some time left for another model. Also,about the implementation, mlpack's architecture is kinda different to Theano'sgraph construction and compilation work, but if you managed to work with Theanoyou shouldn't have a problem.If you like we can discuss any details over the mailing list and brainstorm someideas, discuss an initial class design, etc.I hope this is helpful, let us know if you have any more questions.Thanks,MarcusOn 28 Feb 2017, at 07:06, Сидоров Константин <[email protected]> wrote:Hello Marcus,My name is Konstantin Sidorov, and I am an undergraduate student in Astrakhan State University (Russia). I’m glad to know that mlpack was accepted in GSoC’17 – as a side note, congratulations :)I’m interested to work on project “Augmented Recurrent Neural Networks”. I’m already fairly familiar with deep learning. For example, recently I implemented optimality tightening from “Learning to play in a day” (https://arxiv.org/abs/1611.01606) for the AgentNet (“Deep Reinforcement Learning library for humans”, https://github.com/yandexdataschool/AgentNet). Here is the merged pull request: https://github.com/yandexdataschool/AgentNet/pull/88.As you see, I’m quite familiar with deep learning and Theano. Even though my main field of interest is RL, I would be very interested in doing something new – that is why I’ve chosen “Augmented RNNs”.Of course, on such an early stage I have no detailed plan what (and how) to do – only some ideas. In the beginning, for example, I’m planning to implement NTMs as described in arXiv paper and implement *reusable* benchmarking code (e.g., copy, repeat copy, n-grams).I would like to discuss this project more thoroughly if possible. In addition, this is my first participation in GSoC. So, excuse me in advance if I’ve done something inappropriate.---Best Regards,Konstantin._______________________________________________
mlpack mailing list
[email protected]
http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack--Best Regards,Konstantin.
_______________________________________________ mlpack mailing list [email protected] http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack
