Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16344
@yanboliang Thanks for the detailed review. I have made all changes you
suggested except for the part on the new power link function. Yes, the
canonical link in the Tweedie in general is `1.0 - variancePower`. But other
than the special cases of 'identity', 'log', 'inverse', it is rarely used. The
implementation of the generic power link function is unnecessary in my opinion
for the following reasons:
1. The default link in the Tweedie is overridden in Gaussian, Poisson and
Gamma to the respective canonical link. So when one specifies tweedie with
variancePower = 0, 1, 2, they will have the correct canonical link. I now
allow Tweedie to take link function 'identity', 'log', 'inverse', 'sqrt', which
cover almost all possible links that will be used in practice.
2. If we allow an additional 'linkPower" parameter, we will have two ways
to specify the link: one through 'link' and one through 'linkPower'. This will
be confusing.
3. The implementation of the 'linkPower' is messy with almost no additional
gain. Have to write new powerLink class, fromParams method which determines the
link function based on 'link', 'linkPower' as well as 'family', and also
various checks (consistency between link and linkPower, supported Family and
link, etc). I tried it, and it's very ugly.
If we really want a power link function, can I suggest that we implement it
in a separate PR since it is really not specific to the 'tweedie' family? It
can also be used together with other distributions.
@srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
