GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/11607
[SPARK-12313] [SQL] [RFC] use alt metastore partition pruning api when
possible
## What changes were proposed in this pull request?
Asking the metastore to filter partitions with {{getPartitionsByFilter}}
can silently fail (that is, incorrectly returns no partitions, without
reporting any error) when there are lots of partitions. As a partial
workaround, when the filter criteria is only equality, we can use
{{getPartitions(..., partitionSpec)}} which doesn't appear to have this issue.
## How was this patch tested?
jenkins unit tests. (Note: existing tests are insufficient at this point,
needs more before merging. Also doesn't have a repro of the original problem.)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark SPARK-12313
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11607.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11607
----
commit eaf9005aabb969d9dfe365eaf6ef1cb30aa0818b
Author: Imran Rashid <[email protected]>
Date: 2016-03-09T18:34:38Z
[SPARK-12313] improvement to predicate pushdown with lots of partitions.
Predicate pushdown to filter partitions via string filter expression can
fail when there are lots of partitions. So if possible, use another
metastore api that is more limited, but can't handle inequality
constraints.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
