tustvold commented on issue #8156: URL: https://github.com/apache/arrow-rs/issues/8156#issuecomment-3197212614
> I think this is true if the system treats '-NanandNan` differently for comparisons Arrow-rs and by extension DataFusion consistently use total ordering for floating point comparisons. > FWIW I think that is the case today anyways, but since we have no statistics on Nans DataFusion may incorrectly prune the whole thing DataFusion shouldn't push down floating point predicates, as there are so many buggy implementations (including I believe historical versions of arrow-rs) that the results will likely be incorrect. If DF is pushing down such predicates, it is a bug that should be fixed. FWIW this is the whole motivator for trying to standardise something that is both a clean slate free of buggy writers, whilst also being sufficiently simple that we don't end up with a repeat of the current issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
