[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17446 Thanks! Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17446 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75348/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17446 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17446 **[Test build #75348 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75348/testReport)** for PR 17446 at commit [`8ec57f3`](https://github.com/apache/spark/commit/8ec57f394f3ab0c1fc92545c50e028b7867bb064). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/17446 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17446 `null` should not be simply treated as a `false` literal in filter estimation. Based on the definition, `Not(null)` should return `null`. If we treat `null` as `false`, `Not(null)` will return `1.0`, which is wrong in many cases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17446 **[Test build #75348 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75348/testReport)** for PR 17446 at commit [`8ec57f3`](https://github.com/apache/spark/commit/8ec57f394f3ab0c1fc92545c50e028b7867bb064). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17446 Since this PR does not correctly handle the cases like `Not(Not(null))`, I close this PR at first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17446 @wzhfy `BooleanSimplification ` is unable to get rid of `null`. Boolean operations on null is complex. We need more investigation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/17446 > Ideally, our optimizer rule BooleanSimplification should be able to remove the null literal, like what we did for true and false literals. Here, I think we should just assume our filter estimation does not depend on any other optimizer rules. Sorry I may miss some context, but I prefer enhancing `BooleanSimplification` to deal with null literals. There's no need to complicate estimation logic for those deterministic literals which can be removed by optimizer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17446 Thank you! @ron8hu --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user ron8hu commented on the issue: https://github.com/apache/spark/pull/17446 The logic is straightforward. LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17446 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75291/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17446 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17446 **[Test build #75291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75291/testReport)** for PR 17446 at commit [`1775ca7`](https://github.com/apache/spark/commit/1775ca73e9b2918fdc212fbe517f2e743f048368). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17446 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17446 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75289/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17446 **[Test build #75289 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75289/testReport)** for PR 17446 at commit [`d770dbb`](https://github.com/apache/spark/commit/d770dbbe383d6455682abfd2fc4cc81a1ffef969). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17446 **[Test build #75291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75291/testReport)** for PR 17446 at commit [`1775ca7`](https://github.com/apache/spark/commit/1775ca73e9b2918fdc212fbe517f2e743f048368). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17446 **[Test build #75289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75289/testReport)** for PR 17446 at commit [`d770dbb`](https://github.com/apache/spark/commit/d770dbbe383d6455682abfd2fc4cc81a1ffef969). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17446 Ideally, our optimizer rule `BooleanSimplification` should be able to remove the null literal, like what we did for true and false literals. Here, I think we should just assume our filter estimation does not depend on any other optimizer rules. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17446: [SPARK-17075][SQL][followup] Add Estimation of Constant ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17446 cc @ron8hu @cloud-fan @wzhfy --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
