Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16729
merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
@felixcheung Could you merge this please? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74423/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #74423 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74423/testReport)**
for PR 16729 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #74423 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74423/testReport)**
for PR 16729 at commit
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
Sorry that I forgot to address that comment. Fixed now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #74417 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74417/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74417/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #74417 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74417/testReport)**
for PR 16729 at commit
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
@felixcheung Thanks for the feedback. Made a new commit that
1. change `variancePower` and `linkPower` to `var.power` and `link.power`.
2. use `link = NULL` for tweedie family.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16729
I like the example in this implementation! thanks
yes I think we should name them `var.power` and `link.power`.
---
If your project is set up for it, you can reply to this email and have
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
One other change I could make is to change `variancePower` and `linkPower`
to `var.power` and `link.power` to be consistent with `statmod`. But l would
like to get your feedback on this new
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74243/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #74243 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74243/testReport)**
for PR 16729 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #74243 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74243/testReport)**
for PR 16729 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #74242 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74242/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74242/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #74242 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74242/testReport)**
for PR 16729 at commit
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
@felixcheung OK, new implementation of # 3. Now works in two ways:
1. `family = "tweedie"` + `variancePower` + `linkPower`
2. When `statmod` is available, `tweedie()`
Please
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16729
@actuaryzhang that's true, it's not ideal.
This is somewhat an unusual case for R for several reasons.
In my head the guiding principles are:
- we avoid depending on another
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
@felixcheung If we go with # 3, do we still want to compatibility with
statmod::tweedie? It's confusing to have two different ways of specifying the
same model.
---
If your project is set
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16729
yea - I'm sorry if it was confusing - I was referring to `SparkR::tweedie`
(2 `:`), and not `SparkR:::tweedie` (3 `:`), which was why I wasn't sure that
could be done.
In the past
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
@felixcheung Could you take a look at this new fix when you get a chance?
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
@felixcheung Yes, the SparkR `tweedie` is not exported. See below.
```
model1 <- spark.glm(training, Sepal_Width ~ Sepal_Length + Species,
+ family =
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16729
Thanks for working on this - to clarify, this only works with
`SparkR:::tweedie` (ie. 3 `:`)
As this would be a private implementation?
---
If your project is set up for it, you can reply
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
@felixcheung Sorry for taking so long for this update.
I think your first suggestion makes most sense, i.e., we do not expose the
internal `tweedie`.
When `statmod` is loaded,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73964/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #73964 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73964/testReport)**
for PR 16729 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #73964 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73964/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73960/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #73960 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73960/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #73960 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73960/testReport)**
for PR 16729 at commit
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
@felixcheung Thanks for the discussions. Will work on this in two weeks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16729
hmm.. I'm not sure we could do that because of Apache licensing policy
(unless the owner decides to submit that here).
let me think about this a bit.
---
If your project is set up for
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
@felixcheung The one from statmod will be masked and must be called using
`statmod:tweedie`.
We can copy the whole `tweedie` function from statmod into `SparkR` and
this will avoid the
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16729
ah thanks. so I were to
```
library(statmod)
library(SparkR)
```
could I still access the statmod tweedie function?
ie. does statmod::tweedie still work with R
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72273/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72273 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72273/testReport)**
for PR 16729 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72273 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72273/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72268/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72268 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72268/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72268 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72268/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72261/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72261 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72261/testReport)**
for PR 16729 at commit
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
@felixcheung Great point! Yes, I think it's better to stick with the
statmod syntax and allow the tweedie family to be specified
as`tweedie(var.power, link.power)`. I tried a few ways to allow
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72261 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72261/testReport)**
for PR 16729 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72259 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72259/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72259/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72259 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72259/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72144/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72144 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72144/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72144 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72144/testReport)**
for PR 16729 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72137 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72137/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72137/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72136 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72136/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72136/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72137 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72137/testReport)**
for PR 16729 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72136 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72136/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72132/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72132 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72132/testReport)**
for PR 16729 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72132 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72132/testReport)**
for PR 16729 at commit
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/16729
@felixcheung Thanks so much for your quick and detailed review. I have made
a new commit that removed dependency on `statmod` and fixed the issues you
pointed out. The major change is to add
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72131/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72131 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72131/testReport)**
for PR 16729 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72131 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72131/testReport)**
for PR 16729 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16729
I did look into this... I think it's great if `statmod` is there and we
support it, but I'm concerned that we can't enable this `tweedie` family
without an external dependency and the fact that
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72114/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72114 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72114/testReport)**
for PR 16729 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72114 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72114/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72112/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72112 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72112/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72112 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72112/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72111 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72111/testReport)**
for PR 16729 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72111/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16729
**[Test build #72111 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72111/testReport)**
for PR 16729 at commit
93 matches
Mail list logo
