Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/7446#issuecomment-122027659
Thanks for the fix! There is one thing I think it is better to double
check. Let's say I have a partitioned Parquet table and I select a few columns
from it and then I have predicates on top of it. These predicates have
non-deterministic expressions, a predicate on the partition column, and some
predicates that can be pushed down to parquet table scan (pushed into parquet's
reader). With out fix, will partitioning column get correctly pruned and those
predicates get pushed down to parquet's reader? `ParquetFilterSuite.scala` is
the file for testing parquet filter pushdown. We may need to add tests for
partition pruning. @liancheng Where are our tests for partition pruning?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
