Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/14671
@andreweduffy's comments about this make sense to me. Improving the filters
that are pushed is a good idea, even if we decide to disable Parquet's
row-by-row filtering.
The option to disable row-level filtering,
`parquet.filter.record-level.enabled`, was added specifically for Spark, but
isn't in a released version of Parquet yet. We'll have to wait until 1.9.0 is
released to test with it directly, though it isn't too difficult to filter row
groups and pass those to the file reader if you want to get the same effect.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
