[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user Parth-Brahmbhatt commented on the issue: https://github.com/apache/spark/pull/14655 Closing this PR given its a duplicate at this point. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user lianhuiwang commented on the issue: https://github.com/apache/spark/pull/14655 @wzhfy Yes, I think this is same with SPARK-15616. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/14655 Seems this PR solves similar problem as [SPARK-15616](https://github.com/apache/spark/pull/18193)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14655 @Parth-Brahmbhatt Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user Parth-Brahmbhatt commented on the issue: https://github.com/apache/spark/pull/14655 I will re-evaluate and update or close the PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14655 @Parth-Brahmbhatt Are you still interested in this PR? Our stats refactoring has been finished in the release of 2.2. Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14655 Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user Parth-Brahmbhatt commented on the issue: https://github.com/apache/spark/pull/14655 @gatorsmile not sure if it will simplify much in this case as most of the complexity is in figuring out what partitions can be pruned which I don't think will go away. We will rely on hive metastore instead of hdfs for size calculation whenever partition level stats are stored and available but that part of the code is not really complex. I am fine waiting for the patch to be delivered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14655 How about waiting for a few days until that is delivered? Let us see whether that might simplify your PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user Parth-Brahmbhatt commented on the issue: https://github.com/apache/spark/pull/14655 @gatorsmile not sure if its the same issue. The issue you are pointing at talks about storing the actual partition level stats, which could be used by this PR but until its available we could rely on file system level statistics. Also given this is config driven which is disabled by default it should have no perf impact. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14655 Found a related JIRA: https://issues.apache.org/jira/browse/SPARK-17129 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user Parth-Brahmbhatt commented on the issue: https://github.com/apache/spark/pull/14655 @cloud-fan How do you suggest to change this? I started with Metastore as internally that is the most used datasource and will benefit from partition pruning at planning stage. I am open to any suggestions and will modify the code accordingly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14655 Will this be part of the CBO work? The size estimation or statistics collection is being re-designed for CBO, right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14655 If we gonna do this, I'd like to have a more general approach, which should also work for data source tables. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14655 cc @cloud-fan and @gatorsmile - both are working on refactoring some of these code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14655: [SPARK-16669][SQL]Adding partition prunning to Metastore...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14655 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
