stevomitric opened a new pull request, #59578: URL: https://github.com/apache/spark/pull/59578
### What changes were proposed in this pull request? `percentile_approx` / `approx_percentile` and `histogram_numeric` now reject `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (p in [7, 9]) with `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` in both ANSI modes, instead of implicitly casting them to `DATE`. Neither function lists a nanosecond type in its input `TypeCollection`, so the implicit-cast fallback picks the first store-assignable member, which is `DateType`. For a nanosecond child, `inputTypes` now expects exactly that type, so the analyzer inserts no cast, and `checkInputDataTypes` reports the standard mismatch against the supported types. This is the same error the generic type coercion would give if it didn't pick `DATE`. Other input types, and the error messages for them, are unchanged. Why reject instead of supporting nanos natively: both aggregates compute on doubles. `QuantileSummaries` keeps doubles as samples, and `NumericHistogram` averages double bin centers. A double has a 53-bit mantissa, so around 2020 any numeric encoding has a 256 ns resolution. That covers epoch nanos, epoch micros plus a fraction, and epoch seconds plus a fraction. Only 5 distinct doubles fall within one microsecond, and a single input `2020-01-01 12:00:00.123456789` would come back as `...123456768`. The sketch only compares values, so an order-preserving bit encoding could make it exact, but the nanos range (years 0001-9999) has about 2^68 values at p=9, which no 64-bit key can hold. Exact support would need a sketch with wider keys. Until then a clean error is better than a silently less precise result, and `CAST(c AS TIMESTAMP_NTZ)` / `CAST(c AS TIMESTAMP)` gives the microsecond result explicitly. ### Why are the changes needed? Part of SPARK-56822. Today `SELECT percentile_approx(c, 0.5) FROM VALUES (TIMESTAMP_NTZ '2020-01-01 12:00:00.123456789') t(c)` returns `2020-01-01` typed `DATE` in both ANSI modes, and `histogram_numeric` returns `DATE` bin centers. This is a silent wrong result, which the SPIP's parity criterion rules out. ### Does this PR introduce _any_ user-facing change? Yes, only for the nanosecond timestamp types, which are behind the preview flag `spark.sql.timestampNanosTypes.enabled`. These calls now fail at analysis instead of returning a `DATE`. ### How was this patch tested? - New tests in `ApproximatePercentileSuite` and `HistogramNumericSuite`: NTZ/LTZ at p=7..9, both ANSI modes. - New test in `ApproximatePercentileQuerySuite`: `percentile_approx` and `approx_percentile`, scalar and array percentages, both ANSI modes. - New golden cases in `timestamp-ntz-nanos.sql` and `timestamp-ltz-nanos.sql`: both ANSI modes, plus the explicit-cast workaround. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
