yaooqinn commented on code in PR #45202:
URL: https://github.com/apache/spark/pull/45202#discussion_r1500153887
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala:
##########
@@ -609,8 +609,8 @@ class ParquetFilters(
// Parquet's type in the given file should be matched to the value's type
// in the pushed filter in order to push down the filter to Parquet.
- private def valueCanMakeFilterOn(name: String, value: Any): Boolean = {
- value == null || (nameToParquetField(name).fieldType match {
+ private def valueCanMakeFilterOn(name: String, value: Any, allowNull:
Boolean): Boolean = {
+ (value == null && allowNull) || (nameToParquetField(name).fieldType match {
Review Comment:
Can we move `value == null` to the caller side too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
