IvanVergiliev commented on a change in pull request #24068: [SPARK-27105][SQL]
Optimize away exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#discussion_r288700718
##########
File path:
sql/core/v1.2.1/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala
##########
@@ -63,55 +64,26 @@ private[sql] object OrcFilters extends OrcFiltersBase {
*/
def createFilter(schema: StructType, filters: Seq[Filter]):
Option[SearchArgument] = {
val dataTypeMap = schema.map(f => f.name -> f.dataType).toMap
+ val orcFilterConverter = new OrcFilterConverter(dataTypeMap)
for {
// Combines all convertible filters using `And` to produce a single
conjunction
- conjunction <- buildTree(convertibleFilters(schema, dataTypeMap,
filters))
+ conjunction <-
buildTree(filters.flatMap(orcFilterConverter.trimUnconvertibleFilters))
// Then tries to build a single ORC `SearchArgument` for the conjunction
predicate
- builder <- buildSearchArgument(dataTypeMap, conjunction, newBuilder)
+ builder <- orcFilterConverter.buildSearchArgument(conjunction,
newBuilder)
Review comment:
I'm not sure which loop you're referring to here. The `for-yield` is used
for dealing with the `Option`s that both of these methods can return, and they
can still return Options so we still need the for loop.
We can create a separate function which takes in a trimmed filter and just
handles the building, but I'm not sure how useful that would be. I kind of like
the isolation and slight guarantees that `build` will only be called on a
trimmed filter that having one common `buildSearchArgument` method gives us.
Let me know if you had something else in mind.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
