andygrove opened a new issue, #270:
URL: https://github.com/apache/datafusion-spark/issues/270
Part of #178. Split out of #155, which was closed as a duplicate of #211 and
#212.
### What the functions do
Spark 4.1 added the TIME type, and the hour, minute and second functions
accept it. datafusion-spark's `hour`, `minute`,
`second` and `date_part` don't accept TIME yet.
| Spark SQL | Spark expression (4.1) | Result |
|---|---|---|
| `hour(t)`, `extract(HOUR FROM t)` | `HoursOfTime` | INT |
| `minute(t)`, `extract(MINUTE FROM t)` | `MinutesOfTime` | INT |
| `second(t)` | `SecondsOfTime` | INT (whole seconds) |
| `extract(SECOND FROM t)`, `date_part('SECOND', t)` |
`SecondsOfTimeWithFraction` | DECIMAL(8,6) |
- `hour`, `minute` and `second` are `ExpressionBuilder`s that pick the TIME
expression when the argument is a `TimeType`.
`EXTRACT` and `date_part` go through `TimePart.parseExtractField`, which
accepts `HOUR`/`H`/`HOURS`/`HR`/`HRS`,
`MINUTE`/`M`/`MIN`/`MINS`/`MINUTES` and
`SECOND`/`S`/`SEC`/`SECONDS`/`SECS`. Any other field fails analysis.
- `SecondsOfTimeWithFraction` truncates the fraction to the precision `p` of
`TIME(p)` (`DateTimeUtils.getSecondsOfTimeWithFraction`).
- All of these are `RuntimeReplaceable` and become `StaticInvoke` calls into
`DateTimeUtils`.
The source is
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala`.
### Examples
From `sql/core/src/test/resources/sql-tests/results/time.sql.out` (Spark
4.1.1):
```sql
SELECT extract(HOUR FROM time'23:59:00.987654'); -- 23 (int)
SELECT extract(MINUTE FROM time'00:59:00.987654'); -- 59 (int)
SELECT extract(SECOND FROM time'00:59:00.987654'); -- 0.987654
(decimal(8,6))
SELECT extract(SECOND FROM cast('09:08:01.987654' AS time(0))); -- 1.000000
SELECT extract(SECOND FROM cast('09:08:01.987654' AS time(3))); -- 1.987000
SELECT extract(SECOND FROM cast('09:08:01.987654' AS time(6))); -- 1.987654
SELECT second(TIME'13:25:59.999999'); -- 59 (from the
ExpressionDescription)
```
### Comet implementation
Comet implements `extract(SECOND FROM time)` (`spark_seconds_of_time`). It
takes the TIME value as
`Time64(Nanosecond)` and the precision as a literal, and returns
DECIMAL(8,6):
-
[`datetime_funcs/extract_date_part.rs`](https://github.com/apache/datafusion-comet/blob/3878424d90c23de3c6355a53b293c22822876944/native/spark-expr/src/datetime_funcs/extract_date_part.rs)
### Edge cases to test
- NULL input, midnight, `23:59:59.999999`.
- Every TIME precision from 0 to 6 for `extract(SECOND ...)`.
- All the field aliases above, and an unsupported field such as `DAY` on a
TIME.
### Differences across Spark versions
- TIME exists only in Spark 4.1 and later. In 3.4, 3.5 and 4.0 these
functions take only timestamps (and strings cast to timestamps).
### Implementation notes
- The existing `hour`, `minute` and `second` are in
`crates/spark/src/function/datetime/extract.rs`, and `date_part` is in
`date_part.rs`. Add TIME handling there rather than new functions.
- #38 (`date_part('SECOND', ts)` drops the fractional seconds) is the
timestamp counterpart of the DECIMAL(8,6) result here.
---
_This issue was generated by an LLM (Claude Code) from the Apache Spark
source code, Spark's `.sql.out` golden files and
Comet's source. Please verify the details against Spark before relying on
them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]