[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/11260 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188390037 Looks good, merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188344469 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51882/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188344458 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188344019 **[Test build #51882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51882/consoleFull)** for PR 11260 at commit [`88535ee`](https://github.com/apache/spark/commit/88535ee2e314781ad5e1f478fa53f109d3fc4e3e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188302330 **[Test build #51882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51882/consoleFull)** for PR 11260 at commit [`88535ee`](https://github.com/apache/spark/commit/88535ee2e314781ad5e1f478fa53f109d3fc4e3e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188297160 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188257864 It is weird. Some other PRs like #11344 (just document change) also failed at this SparkR unit tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188212965 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188212970 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51868/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188212474 **[Test build #51868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51868/consoleFull)** for PR 11260 at commit [`88535ee`](https://github.com/apache/spark/commit/88535ee2e314781ad5e1f478fa53f109d3fc4e3e). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188175586 **[Test build #51868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51868/consoleFull)** for PR 11260 at commit [`88535ee`](https://github.com/apache/spark/commit/88535ee2e314781ad5e1f478fa53f109d3fc4e3e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188171128 Failure (at test_mllib.R#133): kmeans ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188170993 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188155543 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51858/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188155538 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188155198 **[Test build #51858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51858/consoleFull)** for PR 11260 at commit [`88535ee`](https://github.com/apache/spark/commit/88535ee2e314781ad5e1f478fa53f109d3fc4e3e). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188130127 **[Test build #51858 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51858/consoleFull)** for PR 11260 at commit [`88535ee`](https://github.com/apache/spark/commit/88535ee2e314781ad5e1f478fa53f109d3fc4e3e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188074445 I think the failed test is caused by updated column pruning rule. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188064958 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188064959 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51842/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188064780 **[Test build #51842 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51842/consoleFull)** for PR 11260 at commit [`b58710f`](https://github.com/apache/spark/commit/b58710f7c25391f31920c01684cfe7dba5299972). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188046410 **[Test build #51842 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51842/consoleFull)** for PR 11260 at commit [`b58710f`](https://github.com/apache/spark/commit/b58710f7c25391f31920c01684cfe7dba5299972). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188043855 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188041663 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-188041666 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51840/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11260#discussion_r53888103
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/JoinOptimizationSuite.scala
---
@@ -92,4 +92,30 @@ class JoinOrderSuite extends PlanTest {
comparePlans(optimized,
analysis.EliminateSubqueryAliases(correctAnswer))
}
+
+ test("broadcasthint sets relation statistics to smallest value") {
+val input = LocalRelation('key.int, 'value.string)
+
+val query =
+ Project(Seq($"x.key", $"y.key"),
+Join(
+ SubqueryAlias("x", input),
+ BroadcastHint(SubqueryAlias("y", input)), Inner, None)).analyze
+
+val optimized = Optimize.execute(query)
+
+val expected =
+ Project(Seq($"x.key", $"y.key"),
+Join(
+ Project(Seq($"x.key"), SubqueryAlias("x", input)),
+ Project(Seq($"y.key"),
+BroadcastHint(SubqueryAlias("y", input))),
+ Inner, None)).analyze
+
+comparePlans(optimized, expected)
+
+assert(optimized.collect {
+ case b @ BroadcastHint(_) if b.statistics.sizeInBytes == 1 => 1
--- End diff --
Because it seems we can't import CanBroadcast into this test, I do update
according to your comment with a little change. Please see if it is appropriate
now. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187831120 Implementation LGTM overall, minor comment on tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11260#discussion_r53824942
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/JoinOptimizationSuite.scala
---
@@ -92,4 +92,30 @@ class JoinOrderSuite extends PlanTest {
comparePlans(optimized,
analysis.EliminateSubqueryAliases(correctAnswer))
}
+
+ test("broadcasthint sets relation statistics to smallest value") {
+val input = LocalRelation('key.int, 'value.string)
+
+val query =
+ Project(Seq($"x.key", $"y.key"),
+Join(
+ SubqueryAlias("x", input),
+ BroadcastHint(SubqueryAlias("y", input)), Inner, None)).analyze
+
+val optimized = Optimize.execute(query)
+
+val expected =
+ Project(Seq($"x.key", $"y.key"),
+Join(
+ Project(Seq($"x.key"), SubqueryAlias("x", input)),
+ Project(Seq($"y.key"),
+BroadcastHint(SubqueryAlias("y", input))),
+ Inner, None)).analyze
+
+comparePlans(optimized, expected)
+
+assert(optimized.collect {
+ case b @ BroadcastHint(_) if b.statistics.sizeInBytes == 1 => 1
--- End diff --
This is a bit of a nit, but I think what you really want to test here is:
```scala
val broadcastChildren = optimized.collect {
case Join(_, CanBroadcast(r), _, _) => r
}
assert(broadcastChildren == 1)
```
With the current test something could break in `Project` (for example) that
would prevent the broadcast from actually happening.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187670946 @rxin I've addressed your comments. Please see if this is appropriate. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187670130 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51763/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187670129 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187669920 **[Test build #51763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51763/consoleFull)** for PR 11260 at commit [`01dfad4`](https://github.com/apache/spark/commit/01dfad40da4c75762feb914de3ab567dacbad3dc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187639017 **[Test build #51763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51763/consoleFull)** for PR 11260 at commit [`01dfad4`](https://github.com/apache/spark/commit/01dfad40da4c75762feb914de3ab567dacbad3dc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187631523 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187630910 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51758/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187630909 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187626059 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187621826 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187621830 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51756/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187611637 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187586197 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51745/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187586195 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187585367 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51744/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187585365 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187583899 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187583900 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51743/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187583894 **[Test build #51743 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51743/consoleFull)** for PR 11260 at commit [`1e35aa3`](https://github.com/apache/spark/commit/1e35aa3771de456424310fd18edbfc1b70149ccf). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187579855 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187579858 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51738/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187579379 **[Test build #51738 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51738/consoleFull)** for PR 11260 at commit [`3360007`](https://github.com/apache/spark/commit/3360007997dbfae84f424fab0308be849af8a234). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187578291 **[Test build #51743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51743/consoleFull)** for PR 11260 at commit [`1e35aa3`](https://github.com/apache/spark/commit/1e35aa3771de456424310fd18edbfc1b70149ccf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187577629 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51742/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11260#discussion_r53743303
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala ---
@@ -137,6 +139,22 @@ class JoinSuite extends QueryTest with
SharedSQLContext {
assert(planned.size === 1)
}
+ test("broadcasthint sets relation statistics to smallest value") {
--- End diff --
yeah, I just can't find appropriate suite for it. I'd use
JoinOptimizationSuite. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187577622 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11260#discussion_r53743187
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -81,11 +81,13 @@ private[sql] abstract class SparkStrategies extends
QueryPlanner[SparkPlan] {
* Matches a plan whose output should be small enough to be used in
broadcast join.
*/
object CanBroadcast {
-def unapply(plan: LogicalPlan): Option[LogicalPlan] = plan match {
- case BroadcastHint(p) => Some(p)
- case p if sqlContext.conf.autoBroadcastJoinThreshold > 0 &&
-p.statistics.sizeInBytes <=
sqlContext.conf.autoBroadcastJoinThreshold => Some(p)
- case _ => None
+def unapply(plan: LogicalPlan): Option[LogicalPlan] = {
+ if (sqlContext.conf.autoBroadcastJoinThreshold > 0 &&
+plan.statistics.sizeInBytes <=
sqlContext.conf.autoBroadcastJoinThreshold) {
--- End diff --
maybe add two more space so it is more clear this is part of the if
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11260#discussion_r53743134
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala ---
@@ -137,6 +139,22 @@ class JoinSuite extends QueryTest with
SharedSQLContext {
assert(planned.size === 1)
}
+ test("broadcasthint sets relation statistics to smallest value") {
--- End diff --
sorry would be great to put this in the optimizer suites, rather than this
file which is an end-to-end suite.
I just took a look at the suites available. I'd rename JoinOrderSuite to
JoinOptimizationSuite, and then put this case there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187574944 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187574948 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51731/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187574635 **[Test build #51731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51731/consoleFull)** for PR 11260 at commit [`a3cadbb`](https://github.com/apache/spark/commit/a3cadbb3cd325840761f1bc5911524b10ba464e2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187552861 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51726/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187552856 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187552314 **[Test build #51726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51726/consoleFull)** for PR 11260 at commit [`3fbdab9`](https://github.com/apache/spark/commit/3fbdab901cc8262310369406c2b48214c586eabc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187551375 **[Test build #51738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51738/consoleFull)** for PR 11260 at commit [`3360007`](https://github.com/apache/spark/commit/3360007997dbfae84f424fab0308be849af8a234). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11260#discussion_r53738663
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ColumnPruningSuite.scala
---
@@ -125,5 +125,31 @@ class ColumnPruningSuite extends PlanTest {
comparePlans(optimized, expected)
}
+ test("Keep broadcast hint when pruning on Join") {
--- End diff --
OK I think it'd make more sense to have this somewhere in some join
optimization suite.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11260#discussion_r53738587
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ColumnPruningSuite.scala
---
@@ -125,5 +125,31 @@ class ColumnPruningSuite extends PlanTest {
comparePlans(optimized, expected)
}
+ test("Keep broadcast hint when pruning on Join") {
--- End diff --
because originally it is proposed to deal with pushdown happened with
column pruning.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187543718 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187543719 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51724/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187543500 **[Test build #51724 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51724/consoleFull)** for PR 11260 at commit [`3fbdab9`](https://github.com/apache/spark/commit/3fbdab901cc8262310369406c2b48214c586eabc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187543222 **[Test build #51731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51731/consoleFull)** for PR 11260 at commit [`a3cadbb`](https://github.com/apache/spark/commit/a3cadbb3cd325840761f1bc5911524b10ba464e2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11260#discussion_r53736870
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ColumnPruningSuite.scala
---
@@ -125,5 +125,31 @@ class ColumnPruningSuite extends PlanTest {
comparePlans(optimized, expected)
}
+ test("Keep broadcast hint when pruning on Join") {
--- End diff --
why is this in column pruning suite?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11260#discussion_r53736860
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -82,7 +82,6 @@ private[sql] abstract class SparkStrategies extends
QueryPlanner[SparkPlan] {
*/
object CanBroadcast {
def unapply(plan: LogicalPlan): Option[LogicalPlan] = plan match {
--- End diff --
we should probably turn this patter matching into just an if/else statement
now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187535784 Indeed. We can now. Will update it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187529295 We can remove the old broadcast hint matching in strategy now, can't we? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187519260 **[Test build #51726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51726/consoleFull)** for PR 11260 at commit [`3fbdab9`](https://github.com/apache/spark/commit/3fbdab901cc8262310369406c2b48214c586eabc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187517033 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187516856 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51723/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187516852 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187515497 **[Test build #51724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51724/consoleFull)** for PR 11260 at commit [`3fbdab9`](https://github.com/apache/spark/commit/3fbdab901cc8262310369406c2b48214c586eabc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187464173 @rxin @marmbrus that is good idea. Let me update this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187431085 I still don't like this approach and find it too hacky. I talked to @marmbrus more offline, and maybe it's easiest to just rely on stats overriding, e.g. when broadcast hint is declared, we set the size of the relation (just in the hint operator) to the smallest possible number (1?). Then it should be robust to pushdowns. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187430709 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187430711 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51672/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187430428 **[Test build #51672 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51672/consoleFull)** for PR 11260 at commit [`ad63367`](https://github.com/apache/spark/commit/ad633679e07041bbf872d0cdaf6401e4defaf82c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187396775 **[Test build #51672 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51672/consoleFull)** for PR 11260 at commit [`ad63367`](https://github.com/apache/spark/commit/ad633679e07041bbf872d0cdaf6401e4defaf82c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187306447 This feels kind of hacky to me. @rxin why doesn't the hint just change the statistics again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187248842 It could be caused by the build failure. The shade JAR is too large now. After merging the latest build, I hit an issue when using mvn. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187186887 Why jenkins can't retest? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187164890 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187098182 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187040244 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187011421 **[Test build #51659 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51659/consoleFull)** for PR 11260 at commit [`03dac45`](https://github.com/apache/spark/commit/03dac455b1a7e3ef586d54d210bca0f762089624). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187011434 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51659/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187011428 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187010823 **[Test build #51659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51659/consoleFull)** for PR 11260 at commit [`03dac45`](https://github.com/apache/spark/commit/03dac455b1a7e3ef586d54d210bca0f762089624). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187005053 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187001554 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51657/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11260#issuecomment-187001550 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11260#discussion_r53585015
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
---
@@ -260,6 +260,20 @@ case class Join(
condition: Option[Expression])
extends BinaryNode with PredicateHelper {
+ private def isBrocastHint(plan: LogicalPlan): Boolean = {
--- End diff --
OK. I've made it as variables to set & copy from other Join operator.
Please take a look if this update is good. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13383][SQL] Keep broadcast hint after c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11260#discussion_r53582665
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
---
@@ -260,6 +260,20 @@ case class Join(
condition: Option[Expression])
extends BinaryNode with PredicateHelper {
+ private def isBrocastHint(plan: LogicalPlan): Boolean = {
--- End diff --
well you don't need to make the hint a parameter of the case class, but
just some field you can set ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
