Hello developers, I'm Karan Desai, an Electrical Engineering Undergraduate at IIT Roorkee. I was following the community since October and initially planned to work on Pytest Migration idea. But on meticulous discussions, it was concluded that the migration task might be short for a three month wide timeline. Besides work is in progress on that.
I particularly found the first project idea appealing, and went about gathering ingredients to make the perfect recipe for summers. Finally I can outline it as stated below. The description was quite short, so I will be happy to include more in it if need be. 1. There's a gradient descent optimizer, but I could not find an optimizer for adaptive learning strategies (I saw a method for adam in MLP though). So adding that can be a part of my project. 2. I looked into benchmarks directory, and checked a comparison of SGD against coordinate descent and ridge regression. Similar type of benching should be done with this new Optimizer/s as well. 3. There's a lack of multinomial logloss as mentioned in description (categorical cross entropy for classification tasks). I can work on adding that as well. As an addition, I can work on KL divergence, poisson and cosine proximity losses, to name a few. In my opinion, these are pretty standard and can be a nice to have. They already exist as metrics, just need to be ported to Cython and used as an optimization objective for linear classifiers. 4. About a tool to anneal learning rate: I suggest a new approach to look at this - as a callback. I searched through the documentation and I could not find this way of handling tidbits during training of models. We should be able to provide a callback to the constructor of a linear model which can do any dedicated job after every epoch, be it learning rate annealing, saving model checkpoint, getting custom verbose output, or as creative as uploading data to server for real time plots on any website. If this gets working in place, we can generalize this to many classes of scikit-learn. As a part of my project, I am planning to enrich scikit-learn to be shipping some ready made callback helpers for easy plug and play. I am still not sure whether this is sufficient for a three months timeline, because I am assuming the review cycles might take slightly longer time because of scikit-learn being such a huge community. As far as the math is concerned, I have searched for some good references, some of which are listed below: 1. First two points will heavily rely on @mblondel's lightning package, and this blog post: http://sebastianruder.com/optimizing-gradient-descent/ 2. For the losses (third point), I have seen the way existing losses are written in cython, as well as in the metrics submodule. That should help a lot. 3. About the fourth point, first of all I would be happy to get some suggestions from the community. Once satisfied, I should implement a very basic prototype with some existing class, maybe convert verbose logging of some class to a callback structure. Will include that in the second draft of proposal which would be a preliminary version of what I shall submit on GSoC website. More about me: 1. Github Profile: https://www.github.com/karandesai-96 2. GSoC 2016 Project: https://goo.gl/mdFZ6m 3. Joblib Contributions: https://git.io/vyMSx 4. Scikit-learn Contributions: https://git.io/vyMSF I'll be eagerly waiting for feedback. Thanks. Regards, Karan Desai, Department of Electrical Engineering, IIT Roorkee, India.
_______________________________________________ scikit-learn mailing list [email protected] https://mail.python.org/mailman/listinfo/scikit-learn
