GitHub user mallman opened a pull request:
https://github.com/apache/spark/pull/15673
[SPARK-17992][SQL] Return all partitions from HiveShim when Hive throws a
metastore exception when attempting to fetch partitions by filter
(Link to Jira issue: https://issues.apache.org/jira/browse/SPARK-17992)
## What changes were proposed in this pull request?
We recently added table partition pruning for partitioned Hive tables
converted to using `TableFileCatalog`. When the Hive configuration option
`hive.metastore.try.direct.sql` is set to `false`, Hive will throw an exception
for unsupported filter expressions. For example, attempting to filter on an
integer partition column will throw a
`org.apache.hadoop.hive.metastore.api.MetaException`.
I discovered this behavior because VideoAmp uses the CDH version of Hive
with a Postgresql metastore DB. In this configuration, CDH sets
`hive.metastore.try.direct.sql` to `false` by default, and queries that filter
on a non-string partition column will fail.
Rather than throw an exception in query planning, this patch catches this
exception, logs a warning and returns all table partitions instead. Clients of
this method are already expected to handle the possibility that the filters
will not be honored.
## How was this patch tested?
A unit test was added.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/VideoAmp/spark-public
spark-17992-catch_hive_partition_filter_exception
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15673.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15673
----
commit c62beda38fe16fb6fe391e125605cf6df90d1e57
Author: Michael Allman <[email protected]>
Date: 2016-10-28T17:35:39Z
[SPARK-17992][SQL] Return all partitions from HiveShim when Hive throws
a metastore exception when attempting to fetch partitions by filter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
