alamb commented on PR #13590: URL: https://github.com/apache/datafusion/pull/13590#issuecomment-2508947757
> > If we value portability, I'd propose that we switch to `libm`, which is what I've implemented in the second commit. > > I think portability is not necessary, and [PostgreSQL](https://www.postgresql.org/docs/17/regress-evaluation.html#REGRESS-EVALUATION-FLOAT-DIFFERENCES) doesn't guarantee it either. I prefer to keep using std, as it should be more mature. I agree with @jonahgao and @findepi -- ensuring exact floating point reproducibility is not something most database systems do, I think due to the (performance) cost of doing so > Floating-point arithmetic is not associative. I think this is the core challenge of why getting reproduceable results on a multi-threaded processing system like DataFusion will be hard without major changes. To ensure the same floating point results requires ensuring the same order of calculation. It implies, for example, that the order processing intermediate aggregate rows must be the same, even if one core is done before another. So TLDR is 1. I think just changing to a different math library is unlikely to be enough 2. If you need reporoducable values I think you might be able to use ordering to achieve it (e.g. order by the group keys in grouing, etc) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
