Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @yanboliang Thanks for the detailed review. I have made all changes you suggested except for the part on the new power link function. Yes, the canonical link in the Tweedie in general is `1.0 - variancePower`. But other than the special cases of 'identity', 'log', 'inverse', it is rarely used. The implementation of the generic power link function is unnecessary in my opinion for the following reasons: 1. The default link in the Tweedie is overridden in Gaussian, Poisson and Gamma to the respective canonical link. So when one specifies tweedie with variancePower = 0, 1, 2, they will have the correct canonical link. I now allow Tweedie to take link function 'identity', 'log', 'inverse', 'sqrt', which cover almost all possible links that will be used in practice. 2. If we allow an additional 'linkPower" parameter, we will have two ways to specify the link: one through 'link' and one through 'linkPower'. This will be confusing. 3. The implementation of the 'linkPower' is messy with almost no additional gain. Have to write new powerLink class, fromParams method which determines the link function based on 'link', 'linkPower' as well as 'family', and also various checks (consistency between link and linkPower, supported Family and link, etc). I tried it, and it's very ugly. If we really want a power link function, can I suggest that we implement it in a separate PR since it is really not specific to the 'tweedie' family? It can also be used together with other distributions. @srowen
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org