Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
@felixcheung Sorry for taking so long for this update.
I think your first suggestion makes most sense, i.e., we do not expose the
internal `tweedie`.
When `statmod` is loaded, users can use `tweedie` directly (from statmod);
otherwise, they can use `SparkR::tweedie` which has the same syntax.
I have made this to work. The following shows it now works both when
statmod is not loaded (using `SparkR:::tweedie`) and when statmod is loaded
(using `tweedie`).
Let me know if there is any other issues. Thanks.
```
training <- suppressWarnings(createDataFrame(iris))
model1 <- spark.glm(training, Sepal_Width ~ Sepal_Length + Species,
family = SparkR:::tweedie(var.power = 1.2, link.power
= 1.0))
summary(model1)$coefficients
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.7009666 0.22970461 7.405017 9.638512e-12
Sepal_Length 0.3436703 0.04518882 7.605206 3.200329e-12
Species_versicolor -0.9703190 0.07090188 -13.685377 0.000000e+00
Species_virginica -0.9852650 0.09129919 -10.791607 0.000000e+00
library(statmod)
model2 <- spark.glm(training, Sepal_Width ~ Sepal_Length + Species,
family = tweedie(var.power = 1.2, link.power = 1.0))
summary(model2)$coefficients
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.7009666 0.22970461 7.405017 9.638512e-12
Sepal_Length 0.3436703 0.04518882 7.605206 3.200329e-12
Species_versicolor -0.9703190 0.07090188 -13.685377 0.000000e+00
Species_virginica -0.9852650 0.09129919 -10.791607 0.000000e+00
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
