[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17918 thanks, merging to master/2.2! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17918 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17918 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76731/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17918 **[Test build #76731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76731/testReport)** for PR 17918 at commit [`d4ab730`](https://github.com/apache/spark/commit/d4ab7307f4fc51ea91f5617b88b4aaa14a85cbab). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17918 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17918 **[Test build #76731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76731/testReport)** for PR 17918 at commit [`d4ab730`](https://github.com/apache/spark/commit/d4ab7307f4fc51ea91f5617b88b4aaa14a85cbab). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17918 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76683/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17918 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17918 **[Test build #76683 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76683/testReport)** for PR 17918 at commit [`2289422`](https://github.com/apache/spark/commit/2289422d20fe4d8012bce7ef26d21c29b2a0635f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17918 **[Test build #76683 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76683/testReport)** for PR 17918 at commit [`2289422`](https://github.com/apache/spark/commit/2289422d20fe4d8012bce7ef26d21c29b2a0635f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/17918 @ron8hu You can look at tpcds q18. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17918 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76664/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17918 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17918 **[Test build #76664 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76664/testReport)** for PR 17918 at commit [`39b72ca`](https://github.com/apache/spark/commit/39b72caff37dc3da1e50beccfbe58f56cedf8007). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ColumnStatsMap(originalMap: AttributeMap[ColumnStat]) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/17918 @ron8hu Yes, but I forgot the query numbers. The case is, first we filter a large part of table A, then join table B on another column. Since ndv of that column is not scaled down, the estimated cardinality of the join is too small. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user ron8hu commented on the issue: https://github.com/apache/spark/pull/17918 I agree that we need to scale down NDV for all the referenced columns in a query if a filter condition reduces the number of qualified rows. Do you find this problem when running tpc-ds benchmark? If so, what queries encounters this issue? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17918 **[Test build #76664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76664/testReport)** for PR 17918 at commit [`39b72ca`](https://github.com/apache/spark/commit/39b72caff37dc3da1e50beccfbe58f56cedf8007). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17918: [SPARK-20678][SQL] Ndv for columns not in filter conditi...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/17918 cc @cloud-fan @gatorsmile @ron8hu --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
