danny0405 commented on code in PR #19646:
URL: https://github.com/apache/hudi/pull/19646#discussion_r3818012386
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/DataSkippingUtils.scala:
##########
@@ -160,11 +160,12 @@ object DataSkippingUtils extends Logging {
Option.empty
})
- // Filter "colA = null"
- // Translates to "colA_nullCount = null" for index lookup
- case EqualNullSafe(attrRef: AttributeReference, litNull @ Literal(null,
_)) =>
+ // Filter "colA <=> null"
+ // Translates to "colA_nullCount = null or colA_nullCount > 0" for index
lookup
Review Comment:
Could we spell this as `colA_nullCount IS NULL OR colA_nullCount > 0` (and
likewise in the other updated comments)? In SQL, `= NULL` evaluates to
unknown—the exact behavior this patch is correcting—while the generated
expression now uses `IsNull`. Using `IS NULL` here would keep the comment
aligned with the implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
