Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-162258164
LGTM. @felixcheung I think the current resolution of not adding `cor` until
we can support it is fine by me. Its better to not mask existing functions
unless we can
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9680
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160810068
**[Test build #46911 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46911/consoleFull)**
for PR 9680 at commit
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160817390
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160815852
second is a git error (seems like having a lot these days?)
`hudson.plugins.git.GitException: Failed to fetch from
https://github.com/apache/spark.git
at
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160819835
**[Test build #46922 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46922/consoleFull)**
for PR 9680 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160806942
**[Test build #46911 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46911/consoleFull)**
for PR 9680 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160810186
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160810185
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160804279
stats::cov name conflict: https://issues.apache.org/jira/browse/SPARK-11886
will open a new JIRA on cor alias once this is merged
---
If your project is
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-16080
thanks, rebased.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160810876
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160810878
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160845233
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160845163
**[Test build #46922 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46922/consoleFull)**
for PR 9680 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160845234
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160627755
@felixcheung, sorry for late response.
Since there is no agreement now, I am fine that we don't add "cor" alias in
this PR. Let's get this PR merged.
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-160627949
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-159493667
@sun-rui ? I"m fine with adding `cor` - are we ok with masking `stats::cor`?
---
If your project is set up for it, you can reply to this email and have your
reply
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9680#discussion_r45259387
--- Diff: R/pkg/R/functions.R ---
@@ -259,6 +259,21 @@ setMethod("column",
function(x) {
col(x)
})
+#'
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-157913206
@shivaram Can we go ahead with this? I think we could consider adding all
character overload for DataFrame functions in a different JIRA.
---
If your project is
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-157975716
As per thisÂ
https://stat.ethz.ch/R-manual/R-devel/library/stats/html/cor.html
It support "vector, matrix or data frame", and it doesn't say a subset of a
data
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-157973061
adding all character overload for DataFrame functions in a different JIRA
is OK.
But for alias of corr(), #9366 only supports inter-column cov and cor of a
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-157977761
Also, since here we are working with 2 columns, by adding a alias `cor` we
will need to create a generic with a different signature that again masks
`stats::cor`,
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9680#discussion_r45167221
--- Diff: R/pkg/R/functions.R ---
@@ -259,6 +259,20 @@ setMethod("column",
function(x) {
col(x)
})
+#' corr
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9680#discussion_r45166927
--- Diff: R/pkg/R/functions.R ---
@@ -259,6 +259,21 @@ setMethod("column",
function(x) {
col(x)
})
+#' corr
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-157513054
any more comment?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9680#discussion_r44974143
--- Diff: R/pkg/R/functions.R ---
@@ -259,6 +259,20 @@ setMethod("column",
function(x) {
col(x)
})
+#'
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9680#discussion_r44973809
--- Diff: R/pkg/R/functions.R ---
@@ -259,6 +259,20 @@ setMethod("column",
function(x) {
col(x)
})
+#'
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9680#discussion_r45019825
--- Diff: R/pkg/R/functions.R ---
@@ -259,6 +259,20 @@ setMethod("column",
function(x) {
col(x)
})
+#'
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-157268953
@sun-rui I updated it. I think it's a bit not as strongly typed as I'd like
but if I add `col2 = "Column"` to signature I get this error:
```
Error in
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9680#discussion_r45018419
--- Diff: R/pkg/R/functions.R ---
@@ -259,6 +259,20 @@ setMethod("column",
function(x) {
col(x)
})
+#' corr
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-157272016
**[Test build #46055 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46055/consoleFull)**
for PR 9680 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-157275713
**[Test build #46055 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46055/consoleFull)**
for PR 9680 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-157275771
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-157275772
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-156957370
these are two different issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9680#discussion_r44899365
--- Diff: R/pkg/R/functions.R ---
@@ -259,6 +259,20 @@ setMethod("column",
function(x) {
col(x)
})
+#' corr
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9680#discussion_r44899428
--- Diff: R/pkg/R/functions.R ---
@@ -259,6 +259,20 @@ setMethod("column",
function(x) {
col(x)
})
+#' corr
Github user NarineK commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-156372445
in R the general formula for correlation is the following:
cor(x, y = NULL, use = "everything", method = c("pearson", "kendall",
"spearman"))
in #9366 is the
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-156484659
So #9366 is for all columns in DataFrame x=y or different x, y DataFrames
And this #9680 is for 2 columns in one DataFrame.
---
If your project is set up for
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-156280168
**[Test build #45789 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45789/consoleFull)**
for PR 9680 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-156278569
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-156278588
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
GitHub user felixcheung opened a pull request:
https://github.com/apache/spark/pull/9680
[SPARK-11715][SPARKR] Add R support corr for Column Aggregration
Need to match existing method signature
You can merge this pull request into a Git repository by running:
$ git pull
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-156331258
I think 9366 is about computing corr or cov matrix whereas this is
computing corr between two columns. They seem to be useful in their own ways.
Also this is
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-156322704
cc @NarineK
I think https://github.com/apache/spark/pull/9366 is a more general form of
this ? If so we could probably just wait for #9366 ?
---
If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-156283280
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-156283275
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9680#issuecomment-156283072
**[Test build #45789 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45789/consoleFull)**
for PR 9680 at commit
50 matches
Mail list logo
