kosiew commented on code in PR #24244: URL: https://github.com/apache/datafusion/pull/24244#discussion_r3801031960
########## datafusion/sqllogictest/test_files/scalar.slt: ########## @@ -1853,6 +1853,24 @@ true false true +# Aggregate CSE preserves the non-nullability of truth predicates +# issue: https://github.com/apache/datafusion/issues/24096 +query IIIII +SELECT + SUM(CASE WHEN b IS TRUE THEN 1 ELSE 0 END), + COUNT(CASE WHEN b IS TRUE THEN 1 END), + SUM(CASE WHEN b IS FALSE THEN 1 ELSE 0 END), + SUM(CASE WHEN b IS NOT TRUE THEN 1 ELSE 0 END), + SUM(CASE WHEN b IS NOT FALSE THEN 1 ELSE 0 END) Review Comment: One small coverage suggestion: it would be useful to include `IS UNKNOWN` and `IS NOT UNKNOWN` in this aggregate-CSE query too. Their physical planning lowers through the same distinctness operators using a NULL literal, so this would cover the two remaining truth-predicate lowering paths. ########## datafusion/physical-expr/src/expressions/binary.rs: ########## @@ -528,7 +528,13 @@ impl PhysicalExpr for BinaryExpr { } fn nullable(&self, input_schema: &Schema) -> Result<bool> { - Ok(self.left.nullable(input_schema)? || self.right.nullable(input_schema)?) + match self.op { + Operator::IsDistinctFrom | Operator::IsNotDistinctFrom => Ok(false), Review Comment: I think we also need to make this change in the logical expression layer. `Expr::BinaryExpr` still derives nullability from its operands in `datafusion/expr/src/expr_schema.rs` (`380-381` and `528-543`), even though `IS DISTINCT FROM` and `IS NOT DISTINCT FROM` never return NULL. With the current change, a direct distinctness expression can therefore have `nullable=true` logically but `nullable=false` physically. Aggregate CSE can extract that expression and recreate the same logical/physical schema mismatch. Could we make the distinctness exception canonical in the logical `nullable` and `to_field` paths as well, and add regression coverage using nullable operands? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
