andygrove opened a new issue, #235:
URL: https://github.com/apache/datafusion-spark/issues/235

   Part of #175 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `timestamp_millis(milliseconds)` returns the `TIMESTAMP` that is 
`milliseconds` ms after `1970-01-01 00:00:00 UTC`.
   The argument must be an integral type (`TINYINT`, `SMALLINT`, `INT`, 
`BIGINT`); the expression uses `ExpectsInputTypes`
   with `IntegralType`, so `DECIMAL`, `FLOAT`, `DOUBLE` and `STRING` are 
analysis errors rather than being cast. The value
   is multiplied by 1000 with `Math.multiplyExact` into microseconds.
   
   The result is always `TIMESTAMP` (LTZ), independent of 
`spark.sql.timestampType`; only its display depends on the
   session time zone. NULL input gives NULL; nullability follows the input. 
Since 3.1.0; the inverse is `unix_millis`
   (already in this crate).
   
   ### Examples
   
   Session time zone `America/Los_Angeles` (as in Spark's docs and golden 
files):
   
   ```sql
   SELECT timestamp_millis(1230219000123);
   -- 2008-12-25 07:30:00.123
   SELECT timestamp_millis(-1230219000123);
   -- 1931-01-07 00:29:59.877
   SELECT timestamp_millis(null);
   -- NULL
   SELECT timestamp_millis(92233720368547758);
   -- java.lang.ArithmeticException: long overflow
   SELECT timestamp_millis(-92233720368547758);
   -- java.lang.ArithmeticException: long overflow
   ```
   
   ### Edge cases to test
   
   - Overflow: `|millis| > i64::MAX / 1000` (9223372036854775) raises 
`ArithmeticException` ("long overflow") with ANSI
     on and off (identical in Spark's ANSI and non-ANSI golden files). Test the 
boundary values on both sides.
   - Negative values: `-1230219000123` is `...:59.877`, i.e. no rounding toward 
zero; `-1` is one millisecond before the
     epoch.
   - `TINYINT` / `SMALLINT` / `INT` / `BIGINT` inputs; `DECIMAL`, `DOUBLE`, 
`STRING` fail with
     `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`.
   - Values far outside years 0001..9999 are valid (any `i64` microsecond value 
is a valid Spark timestamp).
   - The stored value must not depend on `spark.sql.session.timeZone`; display 
across a DST change does.
   - Literal vs column input; dictionary-encoded input.
   
   ### Differences across Spark versions
   
   None found (3.4 through 4.2 have the same evaluation). ANSI mode does not 
affect this function.
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala`,
     `MillisToTimestamp` (extends `IntegralToTimestampBase`, `upScaleFactor = 
MICROS_PER_MILLIS`).
   - DataFusion core `to_timestamp_millis` differs: it returns 
`Timestamp(Millisecond, tz)`, accepts floats, decimals and
     strings, and never fails on overflow (the value is stored as 
milliseconds). Spark's microsecond range is 1000x
     smaller, so values beyond `i64::MAX / 1000` must error. Write a small 
kernel (checked multiply) returning
     `Timestamp(Microsecond, Some(tz))`, with the zone from 
`datafusion.execution.time_zone`.
   - Share code with `timestamp_seconds` / `timestamp_micros` (separate issues) 
in `crates/spark/src/function/datetime/`,
     next to `unix.rs`.
   - Tests: `crates/sqllogictest/test_files/datetime/timestamp_millis.slt` 
exists but every query is commented out
     (ported placeholders). Uncomment and expand it. Document with 
`#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to