Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21623#discussion_r197988392
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala
---
@@ -270,6 +272,29 @@ private[parquet] class ParquetFilters(pushDownDate:
Boolean) {
case sources.Not(pred) =>
createFilter(schema, pred).map(FilterApi.not)
+ case sources.StringStartsWith(name, prefix) if canMakeFilterOn(name)
=>
--- End diff --
What do you think about adding a configuration to control this and set it
true by default? It's basically dependent on an user defined predicate we
manually wrote here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
