etseidl commented on code in PR #15821:
URL: https://github.com/apache/datafusion/pull/15821#discussion_r2098666984
##########
datafusion/physical-optimizer/src/pruning.rs:
##########
@@ -1566,6 +1599,50 @@ fn build_predicate_expression(
return expr;
}
+ // Special handlng for floats. Because current Parquet statistics do not
allow NaN, and
Review Comment:
This block is why we need to get the column ordering info passed down. Here
we know which column is being pruned and with which operation. For floats we
can disallow pruning because the Parquet stats are incomplete. We can also skip
pruning if the stats are not valid because an ordering is not defined for the
type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
