I finally worked it out myself from this research paper,
http://www.stanford.edu/~hastie/Papers/glmnet.pdf
There were two mistakes that I made,
1. We need to add a w_{j} * X[:, j] term to update the residuals. (Eq n 7)
2. When is X is not standardised, there should be a sum of square terms in
the denominator. (Eqn 10 under weighted updates)
So I understood what actually is happening in coordinate_descent.
On Wed, Feb 5, 2014 at 10:32 AM, Manoj Kumar <manojkumarsivaraj...@gmail.com
> wrote:
> I'm sorry it should just be
>
> [image: \frac{\omega_{j}\sum_{i = 1}^n(X_{i}^j)^2 - \alpha + \sum_{i =
> 1}^n (y_{i} - X'\omega)(X_{j}^i)}{\sum_{i = 1}^n (X_{i}^j)^2+ \beta}]
>
>
> in the first equation
>
>
>
>
> On Wed, Feb 5, 2014 at 10:27 AM, Manoj Kumar <
> manojkumarsivaraj...@gmail.com> wrote:
>
>> Hi,
>>
>> I went through the enet_coordinate_descent function in cd_fast.pyx. I
>> have some questions which are noobish but I'll go ahead and ask them anyway.
>>
>> It seems in L176 in each cycle, each omega_j is updated as
>>
>> [image: \frac{\omega_{j}\sum_{i = 1}^n(X_{i}^j)^2 - \alpha + \sum_{i =
>> 1}^n (y_{i} - X'\omega)(X_{j}^i)}{\sum_{i = 1}^n X_{i}^j+ \beta}]
>> ...1]
>>
>> when the term other than alpha in the numerator is greater than, alpha
>> (correct me if I'm wrong)
>>
>>
>>
>> When I went through the wikipedia article, and from my previous
>> knowledge, don't we just do partial derivative of the cost function with
>> respect to omega_{j} and equate it to zero for one cycle of iterations.
>>
>>
>> The cost function is
>> 1 norm(y - X w, 2)^2 + alpha norm(w, 1) + beta norm(w, 2)^2
>> - ----
>> 2 2
>>
>> If we differentiate this with respect to w and equate to zero, shouldn't
>> we get something like
>>
>> [image: -\frac{\alpha + \sum_{i = 1}^n (y_{i} -
>> X'\omega)(X_{j}^i)}{\beta}]
>>
>> I don't understand where I am going wrong
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Regards,
> Manoj Kumar,
> Mech Undergrad
> http://manojbits.wordpress.com
>
--
Regards,
Manoj Kumar,
Mech Undergrad
http://manojbits.wordpress.com
------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
