[GitHub] spark pull request #14285: [SPARK-16649][SQL] Push partition predicates down...

[lianhuiwang]

Github user lianhuiwang closed the pull request at:

https://github.com/apache/spark/pull/14285


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14285: [SPARK-16649][SQL] Push partition predicates down...

[jaceklaskowski]

Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14285#discussion_r71776877
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -632,6 +632,23 @@ class SessionCatalog(
   }
 
   /**
+   * Returns partitions filtered by predicates for the given table, It 
just work for Hive.
--- End diff --

Why do you copy the scaladoc?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14285: [SPARK-16649][SQL] Push partition predicates down...

[jaceklaskowski]

Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14285#discussion_r71776725
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -632,6 +632,23 @@ class SessionCatalog(
   }
 
   /**
+   * Returns partitions filtered by predicates for the given table, It 
just work for Hive.
--- End diff --

Same as above


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14285: [SPARK-16649][SQL] Push partition predicates down...

[jaceklaskowski]

Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14285#discussion_r71776630
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala
 ---
@@ -167,6 +168,21 @@ abstract class ExternalCatalog {
   table: String,
   partialSpec: Option[TablePartitionSpec] = None): 
Seq[CatalogTablePartition]
 
+  /**
+   * Returns partitions filtered by predicates for the given table, It 
just work for Hive.
+   *
+   * The filters Expressions may optionally be provided to filter the 
partitions returned.
+   * For instance, if there exist partitions (a='1', b='2'), (a='1', 
b='3') and (a='2', b='4'),
+   * then the filters (a='1') will return the first two only.
+   * @param db database name
--- End diff --

New line before `@param`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14285: [SPARK-16649][SQL] Push partition predicates down...

[jaceklaskowski]

Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14285#discussion_r71776459
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala
 ---
@@ -167,6 +168,21 @@ abstract class ExternalCatalog {
   table: String,
   partialSpec: Option[TablePartitionSpec] = None): 
Seq[CatalogTablePartition]
 
+  /**
+   * Returns partitions filtered by predicates for the given table, It 
just work for Hive.
--- End diff --

". It just works"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14285: [SPARK-16649][SQL] Push partition predicates down...

[lianhuiwang]

GitHub user lianhuiwang opened a pull request:

https://github.com/apache/spark/pull/14285

[SPARK-16649][SQL] Push partition predicates down into metastore for 
OptimizeMetadataOnlyQuery

## What changes were proposed in this pull request?
SPARK-6910 has supported for pushing partition predicates down into the 
metastore for table scan. So it also should push partition predicates down into 
metastore for OptimizeMetadataOnlyQuery.

## How was this patch tested?
add unit tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lianhuiwang/spark metadata-only-filter

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14285.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14285


commit 4b851056c6e04165683e8ea6370b3e404a688bf2
Author: Lianhui Wang 
Date:   2016-07-20T13:02:11Z

init commit

commit 0773b8219010d4c745b5adde79dedad41ba9a6aa
Author: Lianhui Wang 
Date:   2016-07-20T13:13:08Z

update




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org