HyukjinKwon commented on a change in pull request #23855: [SPARK-26930][SQL]
Tests in ParquetFilterSuite don't verify filter class
URL: https://github.com/apache/spark/pull/23855#discussion_r258871293
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
##########
@@ -114,12 +114,24 @@ class ParquetFilterSuite extends QueryTest with
ParquetTest with SharedSQLContex
new SparkToParquetSchemaConverter(conf).convert(df.schema), pred)
assert(maybeFilter.isDefined, s"Couldn't generate filter predicate
for $pred")
// Doesn't bother checking type parameters here (e.g. `Eq[Integer]`)
- maybeFilter.exists(_.getClass === filterClass)
+ assert(flattenPredicateTree(maybeFilter.get).exists(_.getClass ===
filterClass))
}
checker(stripSparkFilter(query), expected)
}
}
+ private def flattenPredicateTree(filterPredicate: FilterPredicate):
Seq[Object] = {
Review comment:
Hm, but I guess we're a bit unclear if we're going to check nested filters
or only top filter.
Hmmm .. I made another take based upon your branch (see
https://github.com/nandorKollar/spark/pull/1) by disabling problematic
rule(https://github.com/apache/spark/commit/ef77003178eb5cdcb4fe519fc540917656c5d577)
explicitly. You could merge that into your branch so that it can be pushed to
your PR.
If we find no better way, it should be okay to disable that rule for this
test specifically since technically it had to be done together at
https://github.com/apache/spark/commit/ef77003178eb5cdcb4fe519fc540917656c5d577.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
