shrirangmhalgi commented on code in PR #59004:
URL: https://github.com/apache/spark/pull/59004#discussion_r4089027885
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala:
##########
@@ -281,9 +281,18 @@ private[sql] object OrcFilters extends OrcFiltersBase {
.isNull(dataTypeMap(name).fieldName, getType(name)).end())
case In(name, values) if dataTypeMap.contains(name) =>
- val castedValues = values.map(v => castLiteralValue(v,
dataTypeMap(name).fieldType))
- Some(builder.startAnd().in(dataTypeMap(name).fieldName, getType(name),
- castedValues.map(_.asInstanceOf[AnyRef]): _*).end())
+ // SPARK-59605: a NULL in the IN list is inert (`x IN (.., NULL)`
matches the same rows as
Review Comment:
nit: Consider extending the comment to note the Not(In) safety as well:
```
// SPARK-59605: a NULL in the IN list is inert (`x IN (.., NULL)` matches
the same
// rows as IN on the non-null values), and castLiteralValue would NPE on a
null
// literal. Drop NULLs; if none remain (e.g. `IN (NULL)`) there is nothing
safe to
// push. For Not(In), the pushed NOT(IN(non-nulls)) is over-inclusive (reads
extra
// stripes) but safe - Spark re-applies the full predicate after the scan.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
