Hi Ariel.
Thanks for your response and for pointing me to that commit - I didn't
even know this was being actively discussed! I was looking exactly at
that part of the doc-string you refer to in that commit. Now (in
Alex's commit) the docstring is very strange: it looks as though rho
doesn't matter at all! rho=0 is L1 and rho=1 is L1 too!
I just fixed that 5 minutes ago ;)
My understanding is that 'rho' should serve the purpose of what is
referred to as '1-\alpha' in equation 3.54 in chapter 3 of the
Elements of Statistical Learning (this one:
http://www-stat.stanford.edu/~tibs/ElemStatLearn/
<http://www-stat.stanford.edu/%7Etibs/ElemStatLearn/>). That is, when
alpha is high, L2 is penalized and when alpha is low L1 is penalized.
Yes, rho is 1-alpha.
The code itself:
https://github.com/scikit-learn/scikit-learn/blob/9987b61cf87aa8eeecd9c8e2fae6c29599892613/sklearn/linear_model/coordinate_descent.py#L190
Also suggests that high rho would lead to high L1 regularization and
low rho would lead to high L2 regularization, which is consistent with
what I get in my computation. So, I am slowly coming to the conclusion
that the docstring should read (e.g. on line 63):
"[...rho=0...]the penalty is an L2
penalty. For rho = 1 it is an L1 penalty"
Does that make sense to you?
Yes, I think it does. This is also how I understand it.
Can you check if the docstring in current (as of right now ;) master
reflects your understanding?
If you find this inconsistency anywhere else, please let me know.
In SGDClassifier rho=alpha atm but that is a (somewhat) other story.
Cheers,
Andy
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
