andygrove opened a new pull request, #6679:
URL: https://github.com/apache/datafusion-comet/pull/6679
## Which issue does this PR close?
Part of #6385. This is the last item on its checklist, the contributor guide
section on which Spark rule an expression follows and how to test for it. It
doesn't close the epic.
**Stacked on #6447**, because the page describes where #6447 puts comparison
normalization. Review 953a4668b. I'll rebase onto `main` and mark this ready
once #6447 lands.
## Rationale for this change
Most of the fixes under #6385 came from the same two gaps. A Spark function
inherits its float equality from whichever Java API its implementation calls,
and there are five of them. And a test that only uses canonical NaNs passes on
Apple Silicon, while the same query fails on x86-64, where arithmetic produces
NaNs with the sign bit set. None of that was written down for contributors, so
each new expression had to rediscover it.
## What changes are included in this PR?
- A new page, `contributor-guide/floating_point.md`, under Project
Architecture after Timezone Handling. It covers:
- Spark's five rules and which functions use them, and the releases in
which `collect_set`, `mode`, `array_distinct`/`array_union` and map
construction changed rules.
- How Arrow and DataFusion differ.
- Where Comet applies each rule: the `float_semantics` helpers, key
normalization in the planner, the comparison builder, `IN`, hashing, and the
Scala version policies.
- How to choose the rule for a new expression, guidelines for native code,
and how to test, including how to make a sign-bit NaN that survives a Parquet
round trip and how `CometFloatSemanticsSuite`'s known gaps work.
- A testing tip in `adding_a_new_expression.md` that points to it.
## How are these changes tested?
Docs only. prettier 3.9.9 `--check` passes on the three files. The files,
helpers and fixtures the page names exist at this commit, and so do the pages
it links to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
