andygrove opened a new pull request, #334:
URL: https://github.com/apache/datafusion-spark/pull/334
## Which issue does this PR close?
None.
**Stacked on #333.** That PR adds full-precision float output to the test
harness, which these tests rely on, and it is the first commit here. Review
only the second commit, `feat(spark): add Spark-compatible ln, log, log10, log2
and log1p`.
## Rationale for this change
These names currently resolve to DataFusion's built-ins, which differ from
Spark:
| Query | Before | Spark 4.2.0 |
|---|---|---|
| `ln(0)`, `log10(0)` | `-Infinity` | `NULL` |
| `ln(-1)`, `log2(-1)` | `NaN` | `NULL` |
| `log(2.0)` (one argument) | `0.301…` (base 10) | `0.6931471805599453`
(natural log) |
| `log(0, 1)` | `-0` | `NULL` |
| `log(3, 81)` | `4` | `4.000000000000001` |
| `log1p(x)` | function not found | `ln(1 + x)` |
## What changes are included in this PR?
- `math/log.rs`: `ln`, `log` (one or two arguments), `log10`, `log2` and
`log1p` over DOUBLE, with numeric arguments cast to DOUBLE. Like Spark's
`UnaryLogExpression` and `Logarithm`, an input at or below the asymptote (0, or
−1 for `log1p`) returns NULL, and so does a base ≤ 0. NaN is not below the
asymptote, so it returns NaN. `log2` is `ln(x) / ln(2)`, as in Spark, so
`log2(2^29)` is `29.000000000000004`. `log(1, x)` divides by zero and returns
±Infinity or NaN.
- `math/fdlibm.rs`: Spark computes these with `java.lang.StrictMath`, which
the Java spec requires to return exactly fdlibm's results. Rust's `f64::ln`
calls the platform libm. That differs from fdlibm in the last bit on about 2%
of inputs (about 5% for `log10`), and differently on macOS and Linux. The
`libm` crate follows musl, which simplified fdlibm's final steps and also
rounds differently. This file is a direct translation of fdlibm 5.3's
`e_log.c`, `e_log10.c` and `s_log1p.c` from C to Rust. Constants are written as
the bit patterns given in fdlibm's comments.
- **Licensing:** the code is translated from Sun's original fdlibm C
sources. Nothing comes from OpenJDK. Sun's notice ("Permission to use, copy,
modify, and distribute this software is freely granted, provided that this
notice is preserved") is kept verbatim in the file and added to `LICENSE.txt`.
Reviewers, please confirm this is acceptable under ASF policy.
- Each function is documented with `#[user_doc]`.
## Are these changes tested?
- **Ground truth:**
- The fdlibm translation returns the same bits as `StrictMath.log`,
`log10` and `log1p` on 2,000,000 random inputs: random bit patterns, [0, 100),
(−1, 1) and integers. The `StrictMath` outputs were identical on JDK 17 and JDK
21.
- The NULL/NaN semantics and the exact values (`log(3, 81)`, `log(10,
1000) = 2.9999999999999996`, `log2(2^29)`, `ln(2)`) were confirmed by
evaluating Spark 4.2.0's `Log`, `Log2`, `Log10`, `Log1p` and `Logarithm` from
the catalyst jar, through both the interpreted and generated-code paths.
- **Unit tests:** pin the `StrictMath` bits at inputs where Rust's std
differs, plus special values.
- **`.slt` files:** the five placeholder files are now real tests, covering
integer, float and decimal inputs, NULL, NaN, ±Infinity, subnormals, values ≤
0, scalar and array paths, and inputs where `f64::ln`, `f64::log10` and
`f64::ln_1p` give a different last digit.
- `cargo fmt --all -- --check` and `cargo clippy --all-targets
--all-features -- -D warnings`: clean
- `cargo test -p datafusion-spark`: 338 passed
- `cargo test -p datafusion-spark-sqllogictest`: 249 of 249 files passed
## Are there any user-facing changes?
Yes. `ln`, `log`, `log10` and `log2` now follow Spark instead of
DataFusion's built-ins, and `log1p` is new.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]