Hello Shogun, The following is a discussion between Wu Lin (yorkerlin) and myself (ialong) concerning the GSoC 2016 project Large-Scale Gaussian Processes. It contains a detailed overview of the waypoints of the projects by Wu (latest message).
I have condensed the conversation in a single email to avoid spamming the list. Thanks, Alessandro > On 29 Mar 2016, at 19:57, yorker lin <[email protected]> wrote: > > > Hi Alessandro, > > We are always looking for GP people who want to contribute to Shogun's GP :). > As you can tell from the deep learning conference, > http://www.iclr.cc/doku.php?id=iclr2016:main#accepted_papers_conference_track > <http://www.iclr.cc/doku.php?id=iclr2016:main#accepted_papers_conference_track>, > GP is going to become popular. > Since inference for co-variance matrix is non-trivial, libraries which > support inference for (deep) GP are limited. > Which is more, the time complexity of GP inference is still non-linear in > general. > It is a good chance for us to offer a "black-box" library for deep GP > learning. What is more, we might become leaders in deep GP learning. :) > > > The short-term goal is: > We want to clean up/improve the existing codes and refactor the existing > framework so that the framework can be easily extended to deep GP. > In order to do so, We have to learn lessons from the deep learning community. > How to design a flexible framework for deep GP? We can learn from GPFlow and > SkFlow. > In order to achieve this, I will work full-time this summer for Shogun's GP. > :) > It will be great if you have related experience to work with me to achieve > this goal. > > About the latter, I wanted to ask whether you think it will be necessary to > alter the general structure significantly. If I understood correctly, the > main focus of the project should be to clean up the existing code and, in > particular, to separate the linear algebra implementation from the base > classes. So, perhaps, it would be sufficient to only add classes to the > hierarchy that deal with the algebraic subroutines (as well as more general > ones like an ELBO base class). > Yes, we want to > 1. separate the linear algebra implementation > 2. separate optimizers > 3. refactor parameter framework and model selection > 4. separate samplers (optional) > 5. design a layer-wise framework for GPLVM and deep GP > 6. use auto-diff > > You can do step 1 and step 2. If possible, you could do step 3 > I will help with you to do step 1,2 and 3. > Meanwhile, we can jointly design a framework for step 5 and step 6. > This project pays more attention on software engineering than ML algorithms. > Do you think whether you want to do this project? > Can you show us your passionate? (eg, small PRs) > > The long-term goal is: > We want to bridge the computational gap in GP so that the ML/DL community can > do large-scale learning using (deep) GPs. > To achieve this goal, we are looking for long-term developers beyond GSoC. :) > > It could be you! > > > Best, > Wu > > > > On Tue, Mar 29, 2016 at 12:35 PM, Alessandro Davide Ialongo > <[email protected] <mailto:[email protected]>> wrote: > Hi Wu, > > Thank you very much for you reply. > Regarding your questions: > >> Do you have any experience about framework design? >> If no, can you learn from other libraries in a short time? > > I am currently contributing to the development of a data processing pipeline > for the company I work with. This is a pretty broad and complicated > framework, although I am not the only/primary developer. I’d say in general I > am a fast learner and I already have some familiarity with the other > libraries, so I will be able to learn in a short time (and learning about > this is the main reason I am so interested in this project). > >> About varaitional inference for GP >> Do you understand variational inference for GP? If no, can you learn it in a >> short time? > > I have read many papers on the subject (all papers by the Cambridge group, > the fundamental papers by Titsias, Neil Lawrence and most from the Oxford GP > group). I am also familiar, in general, with variational inference from my > Master’s and subsequent research year. I worked with a variational inference > toolbox (in MATLAB) while I was at Gatsby, so I also have familiarity with > implementing variational inference. > Approximate inference methods (and variational inference in particular) is > what I was offered a place to work on at Cambridge, starting in October. > > > Regarding next steps, I will work on benchmarking (if you think that’s > worthwhile) and try to come up with a simple class diagram. > About the latter, I wanted to ask whether you think it will be necessary to > alter the general structure significantly. If I understood correctly, the > main focus of the project should be to clean up the existing code and, in > particular, to separate the linear algebra implementation from the base > classes. So, perhaps, it would be sufficient to only add classes to the > hierarchy that deal with the algebraic subroutines (as well as more general > ones like an ELBO base class). > > Thank you for sharing your GSoC experience, that’s very helpful. I will also > try to come up with a more impressive PR, to give you a better idea of > myself! > > Alessandro > >> On 29 Mar 2016, at 05:41, yorker lin <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Alessandro >> >> Sorry for the late respond. In general, you are a good candidate. >> As you can tell from the project, we focus on more software engineering this >> year. >> >> If possible, can you send us some small PRs to show your software >> engineering skill and understanding of variational inference? >> You have still two weeks do that. >> >> It will be great, if you can design a *simple* class diagram for variational >> methods in GP. >> You can take a look at this page. >> http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CKLInferenceMethod.html >> >> <http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CKLInferenceMethod.html> >> >> You can give us a draft of the diagram. If you are accepted, you can refine >> the diagram many times. >> >> About software engineering >> Do you have any experience about framework design? >> If no, can you learn from other libraries in a short time? >> >> About varaitional inference for GP >> Do you understand variational inference for GP? If no, can you learn it in a >> short time? >> >> >> Let me tell you about my story about GSoC. I worked for Shogun during GSoC >> 2014. >> At the beginning, I was new to GP. I read the GPML book and Shogun's GP >> codes. >> Finally, I submitted my first PR about Laplace method for GP classification >> using LBFGS before the deadline of application. >> The method is still the fastest method for full GP classification. >> The PR might give mentors a good impression. >> >> >> BTW, my PR is not in a entrance task. You do not require to do similar >> thing. >> >> >> >> On Mon, Mar 28, 2016 at 10:43 AM, Alessandro Davide Ialongo >> <[email protected] <mailto:[email protected]>> wrote: >> Dear Wu, >> >> I just wanted to say I submitted a final GSoC proposal on Friday, slightly >> edited in an attempt to take into account feedback from you and Heiko. Thank >> you for all your comments, I really appreciate your help through the >> application process. >> >> However, I fear you might still find the proposal to be light on details, >> and I certainly feel that I could have a clearer picture on what the exact >> waypoints should be for the project, and exactly which are the essential >> deliverables. >> >> I hope you are still considering to do the project with me and if there’s >> anything else you’d like me to do in the short term to give you a better >> impression of my commitment please let me know. On my part I would of course >> be very keen to discuss the project with you in more detail, but I >> understand you might be very busy. >> >> In any case, thank you for you help. >> >> Alessandro >> (ialong) >> > > > > > -- > best, > > wu lin
