Max Gekk created SPARK-57528:
--------------------------------
Summary: Support nanosecond-precision timestamps in the
`unix_timestamp` function
Key: SPARK-57528
URL: https://issues.apache.org/jira/browse/SPARK-57528
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
h3. Background
Part of the SPARK-56822 umbrella. {{unix_timestamp(timestamp)}} (and the no-arg
form)
returns the number of whole seconds since the Unix epoch as {{BIGINT}}. It
currently accepts
{{TimestampType}}, {{DateType}} and strings, but not the new
nanosecond-precision timestamp
types.
h3. Proposal
Allow {{unix_timestamp}} to accept nanosecond-precision timestamps
({{TIMESTAMP_LTZ(p)}} / {{TIMESTAMP_NTZ(p)}}, i.e. {{AnyTimestampNanoType}}) in
its
timestamp-argument form. The result stays {{BIGINT}} seconds; sub-second digits
are
truncated, consistent with current behavior.
* Value: {{Math.floorDiv(epochMicros, MICROS_PER_SECOND)}}, read from
{{TimestampNanosVal}}.
* No overflow concern (result is seconds-granular).
{code:sql}
SELECT unix_timestamp(TIMESTAMP_LTZ '2008-12-25 15:30:00.123456789');
-- 1230219000
{code}
h3. Implementation notes
* Extend the input type check of the {{UnixTime}} base ({{UnixTimestamp}} /
{{ToUnixTimestamp}})
to include {{AnyTimestampNanoType}}.
* In the timestamp branch, read {{epochMicros}} from {{TimestampNanosVal}}
instead of a plain
{{Long}}.
* Interpreted + codegen paths.
h3. Out of scope
* The string-parsing overload {{unix_timestamp(str, fmt)}} producing nanosecond
precision -
tracked separately under Parsing/Formatting (P1).
h3. Acceptance criteria
* {{unix_timestamp}} accepts {{TIMESTAMP_LTZ(p)}} / {{TIMESTAMP_NTZ(p)}} and
returns the
correct whole-second {{BIGINT}}.
* Existing micros / date / string behavior unchanged.
* Unit tests + golden files added.
Parent: SPARK-56822
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
