stevomitric opened a new pull request, #59781: URL: https://github.com/apache/spark/pull/59781
### What changes were proposed in this pull request? This PR makes the exact percentile aggregates `percentile`, `median`, `percentile_cont` and `percentile_disc` accept nanosecond-precision timestamps (`TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)`, p in [7, 9]) and return the input type, family and precision. It's part of the nanosecond timestamp SPIP (SPARK-56822). - `AnyTimestampNanoType` is appended last to the input `TypeCollection`. A nanos input matches it directly, so no cast is inserted. Because it's last, the implicit-cast fallback for every other input type picks the same member as before. - `percentile_disc` returns an input value, so the result is exact. - `percentile`, `median` and `percentile_cont` interpolate `lower + fraction * (higher - lower)` exactly in epoch nanoseconds using `java.math.BigDecimal`, then floor the result to the input precision (100 ns for p=7, 10 ns for p=8). A double can't do this: epoch nanoseconds need about 61 bits, and the 0001..9999 span overflows a long. The numeric, interval and TIME paths are unchanged. Design decision for reviewers: an interpolated value that falls between two grid points is floored toward -inf. For example, the median of `.000000001` and `.000000002` at p=9 is `.000000001`. Flooring matches how nanos values are already stored at a lower precision, and it keeps the result between the two neighboring values for both ascending and descending order. The TIME and interval paths truncate toward zero with `.toLong`. That's the same thing for TIME, which is never negative, and differs only for pre-epoch timestamps. Round-half-even would also work, but it would differ from the floor that casts to a lower nanos precision already apply. ### Why are the changes needed? Since SPARK-60335 stopped the implicit narrowing, these functions reject a nanos timestamp with `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` in both ANSI modes; before that, ANSI silently cast it to `TIME(6)`. The SPIP asks for the supported functions to work at nanosecond precision, and the exact percentile family can do so without losing any digits. ### Does this PR introduce _any_ user-facing change? Yes, for the nanosecond timestamp preview types only (`spark.sql.timestampNanosTypes.enabled`). These functions now return a nanos timestamp of the input type instead of a type-mismatch error. The `UNEXPECTED_INPUT_TYPE` message for other unsupported inputs now also lists the nanos types. Microsecond timestamps behave as before. ### How was this patch tested? - `PercentileSuite`: result types, exact interpolation (sub-microsecond, pre-epoch, the 0001..9999 range, p=7/8 flooring, descending order), `percentile_disc` (current and legacy calculation), and serialized buffer merge. - `TimestampNanosFunctionsAnsiOnSuite` / `AnsiOffSuite`: SQL and Column API results and types for NTZ/LTZ at p=7/8/9 over multiple partitions. - `timestamp-ntz-nanos.sql` / `timestamp-ltz-nanos.sql` golden files, including window, frequency and ANSI-off cases; the four functions move out of the SPARK-60335 rejection block. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
