> For n_features > n_samples, I believe that coordinate descent is > faster in the dual. A primal coordinate descent needs to optimize one > w_i at a time. Therefore, if your data is high dimensional it can take > time. Liblinear implements shrinking to avoid revisiting some > coordinates. Maybe the greedy selection of coordinates you mention can > also help. But then, can it be called cyclic?
with the sparse penalty it's faster in the primal as most w_i are zero and you end up working on a limited set of features. >> I would indeed like to see a fast coordinate descent solver for logistic >> regression. I am more interested in the l1 penalty, but the l2 penalty is >> also useful. Multinomial loss could fall in such work. >> >> For such contribution to be actually useful, I'd like the code to be >> really fast with large n_features: we don't need a solver that doesn't >> scale to real problem. I am not an expert, but I think that a reference >> that I recently mentionned could be useful: >> >> http://www.jmlr.org/papers/volume11/yuan10c/yuan10c.pdf > > Note that the coordinate decent newton (CDN) algorithms for Logistic > Regression and L2-SVM mentioned in that paper are already in liblinear > and hence in scikit-learn :) yes but not for the multinomial. The good way is probably to use the same strategy but that's a lot of work to patch or rewrite part of liblinear. Regarding adding a BFGS implementation, we've always being reluctant to commit too naive implementation but I don't want to be dogmatic on this if it can be useful to somebody. Maybe we should list on the wiki all the gist that follow the scikit API and that might be useful. WDYT? Alex ------------------------------------------------------------------------------ Virtualization & Cloud Management Using Capacity Planning Cloud computing makes use of virtualization - but cloud computing also focuses on allowing computing to be delivered as a service. http://www.accelacomm.com/jaw/sfnl/114/51521223/ _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
