Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15673
> Could we enable this fallback only when the conf is set to false?
Otherwise, it might mask legitimate bugs.
Certainly, but my intent with this PR is to prevent a (painful and
confusing) regression for some Hive users of Spark 2.1 which can occur, because
Spark 2.1 enables our new partition pruning implementation by default. I
mentioned one case where this will happen, but we can't be sure this is the
only case. If we make the conditions under which we use a fallback too narrow,
we are assuming that other configurations of Hive are compatible with partition
pruning outside of the specific conditions we check. I think that's a bit too
risky. In fact, before submitting this PR I had written the catch block to
catch and fall back for all types of `Exception`. What I ended up with here is
a middle ground.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
