[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15493 Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/15493 I will take that and add my note. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user srinathshankar commented on the issue: https://github.com/apache/spark/pull/15493 Created https://issues.apache.org/jira/browse/SPARK-18013 @felixcheung @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15493 cc @felixcheung We still need this. I'm going to create an upstream ticket. Can one of you take it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15493 Why not just introduce a crossJoin function in R, similar to Python/Scala/Java? We don't want to change the default join type, because it is still valid to run an inner join by specifying a predicate later using the filter operator. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/15493 That's a great point. Currently R is the same as Python in that when `joinExpr` is `NULL` (R) or `on` is `None` (Python), CrossJoin is assumed. (Python [here](https://github.com/apache/spark/pull/15493/files#diff-6fc344560230bf0ef711bb9b5573f1faR650)) Problem is by default `joinExpr = NULL` (R) and `on = None` (Python) - one approach is to change both defaults, so that when omitted, they don't default to CrossJoin. Another approach is to additional ask for `joinType` or `how` of a new `cross_join` - but this could be a bigger change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15493 The issue is that we want to prevent users from shooting themselves in the foot, i.e. we want to avoid accidental cross joins. The idea is unless the user explicitly says crossJoin, we should disallow crossjoins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/15493 @rxin In R, CrossJoin is the default when expr is empty (https://github.com/apache/spark/blob/master/R/pkg/R/DataFrame.R#L2304) I reviewed the code and documentation I think it is sufficient --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15493 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15493 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66998/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15493 **[Test build #66998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66998/consoleFull)** for PR 15493 at commit [`8b60ef2`](https://github.com/apache/spark/commit/8b60ef2c6d5d42c4b9202d6743ce606d04630ab5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15493 cc @felixcheung do we need some change for R? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15493 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15493 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15493 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66996/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15493 **[Test build #66996 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66996/consoleFull)** for PR 15493 at commit [`a450857`](https://github.com/apache/spark/commit/a4508579a4b4d31ff8b8a8d71e241d38a4fe2cda). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15493 LGTM pending Jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15493 **[Test build #66998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66998/consoleFull)** for PR 15493 at commit [`8b60ef2`](https://github.com/apache/spark/commit/8b60ef2c6d5d42c4b9202d6743ce606d04630ab5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15493 **[Test build #66996 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66996/consoleFull)** for PR 15493 at commit [`a450857`](https://github.com/apache/spark/commit/a4508579a4b4d31ff8b8a8d71e241d38a4fe2cda). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15493 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66982/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15493 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15493 **[Test build #66982 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66982/consoleFull)** for PR 15493 at commit [`16c0842`](https://github.com/apache/spark/commit/16c084217b915f194956b73b7abf80789a82aef8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15493 **[Test build #66982 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66982/consoleFull)** for PR 15493 at commit [`16c0842`](https://github.com/apache/spark/commit/16c084217b915f194956b73b7abf80789a82aef8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15493 **[Test build #66981 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66981/consoleFull)** for PR 15493 at commit [`9b4d995`](https://github.com/apache/spark/commit/9b4d995fa5e7ca0d9636b708fcec1088054c1022). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15493 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15493 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66981/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15493 **[Test build #66981 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66981/consoleFull)** for PR 15493 at commit [`9b4d995`](https://github.com/apache/spark/commit/9b4d995fa5e7ca0d9636b708fcec1088054c1022). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15493: [SPARK-17946][PYSPARK] Python crossJoin API similar to S...
Github user srinathshankar commented on the issue: https://github.com/apache/spark/pull/15493 cc @sameeragarwal @hvanhovell @davies @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
