leoyvens commented on PR #13590: URL: https://github.com/apache/datafusion/pull/13590#issuecomment-2508063268
There are myths and truths to floating-point reproducibility across platforms. Some facts I've gathered while working on this: 1. f32 and f64 in Rust follow IEEE-754. 2. The IEEE-754 basic arithmetic operations are reproducible. 3. All modern hardware correctly implements IEEE-754. 4. Floating-point arithmetic is not associative. 5. The floating point functions fall under "Recommended Operations" under IEEE-754. These seem to be non-compliant across libc and hardware implementations. For many real-world DataFusion use cases, floating-point operations are reproducible. In my use case, I care about reproducibility so it's not just a tests problem, it's a product problem that I'd like the tests to cover. Epsilon comparison should be used for any test that surfaces a concrete reproducibility problem due to point 4, non-associativity. So far, the only such situation surfaced by local and CI testing was the tests for the `regr_*` functions. On the `libm` question, I'm proposing it because it solves a problem for me. Otherwise I might have to redefine portable versions of those UDFs for my application. Maturity is a valid question. `libm` is under the `rust-lang` organization, it is used by rustc for WASM and has the goal of eventually being made part of `std::core`. If `libm` conflicts with other DataFusion use cases, or if general prudence dictates that we stick with `std`, I will follow the decision made by the DataFusion maintainers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
