dongjoon-hyun opened a new pull request, #59571:
URL: https://github.com/apache/spark/pull/59571
### What changes were proposed in this pull request?
This PR makes `Literal.sql` pad the fractional seconds of a `TIME(p)`
literal with `p` in `[7, 9]` back to exactly `p` digits. It reuses the existing
private `padToNanosPrecision` helper, which already does the same for the
nanosecond-precision timestamp literals (SPARK-57339, #56494).
`TIME(0)`-`TIME(6)` literals and `Literal.toString` (display-only, trailing
zeros stripped) are unchanged.
### Why are the changes needed?
`Literal.sql` renders a `TIME` literal through `FractionTimeFormatter`,
which strips trailing fractional zeros. Since SPARK-57551 (#56622), the parser
derives `TIME(7)`-`TIME(9)` from the number of fractional digits of a `TIME`
literal and gives `TIME(6)` to a literal with 6 or fewer digits. So a `TIME(9)`
literal whose value ends with zeros, e.g. `12:00:00.100000000`, is rendered as
`TIME '12:00:00.1'` and parses back as `TIME(6)`.
This causes wrong query results. By default
(`spark.sql.legacy.parameterSubstitution.constantsOnly=false`),
`spark.sql(sqlText, args)` and `EXECUTE IMMEDIATE ... USING` substitute each
argument into the SQL text as `Literal.sql` and parse the result. So a
`TIME(9)` argument becomes `TIME(6)`, and a string compared or combined with it
is cast to `TIME(6)`, i.e. truncated to microseconds. For example, with `val t
= expr("TIME '12:00:00.100000000'")`:
| Query | Before | After |
| --- | --- | --- |
| `spark.sql("SELECT :t = '12:00:00.100000001'", Map("t" -> t))` | `true` |
`false` |
| `spark.sql("SELECT ? < '12:00:00.100000001'", Array(t))` | `false` |
`true` |
| `spark.sql("SELECT s FROM VALUES ('12:00:00.100000001'), ('12:00:00.1') AS
t(s) WHERE s = :t", Map("t" -> t))` | `12:00:00.100000001`, `12:00:00.1` |
`12:00:00.1` |
| `EXECUTE IMMEDIATE 'SELECT ? = \'12:00:00.100000001\'' USING TIME
'12:00:00.100000000'` | `true` | `false` |
| `spark.sql("SELECT CASE WHEN id = 0 THEN :t ELSE '12:00:00.123456789' END
FROM range(1, 2)", Map("t" -> t))` | `12:00:00.123456` (`TIME(6)`) |
`12:00:00.123456789` (`TIME(9)`) |
In addition, `Literal.sql` is used for the `EXISTS_DEFAULT` column metadata,
the auto-generated column names, and the values in error messages.
`EXISTS_DEFAULT` keeps the value because it is cast back to the column type,
but the column names and the error messages drop the digits written in the
literal.
### Does this PR introduce _any_ user-facing change?
Yes. This fixes the wrong results above. In addition, the auto-generated
column names and the values in error messages keep all `p` fractional digits of
a `TIME(7)`-`TIME(9)` literal.
```scala
spark.sql("SELECT TIME '12:00:00.100000000'").schema.simpleString
spark.sql("SELECT CAST(TIME '23:59:59.000000000' AS TINYINT)").collect()
```
**BEFORE**
```
struct<TIME '12:00:00.1':time(9)>
[CAST_OVERFLOW] The value TIME '23:59:59' of the type "TIME(9)" cannot be
cast to "TINYINT" due to an overflow. ...
```
**AFTER**
```
struct<TIME '12:00:00.100000000':time(9)>
[CAST_OVERFLOW] The value TIME '23:59:59.000000000' of the type "TIME(9)"
cannot be cast to "TINYINT" due to an overflow. ...
```
The behavior changes only for `TIME(7)`-`TIME(9)` literals whose values end
with fractional zeros. This is not a user-facing change compared to the
released Apache Spark versions because `TIME(7)`-`TIME(9)` was added by
SPARK-57551, which exists only in the unreleased `master`, `branch-4.x`, and
`branch-4.3` (Apache Spark 4.3.0 release candidates,
[v4.3.0-rc1](https://github.com/apache/spark/releases/tag/v4.3.0-rc1)
(2026-08-31) and
[v4.3.0-rc2](https://github.com/apache/spark/releases/tag/v4.3.0-rc2)
(2026-10-12)).
### How was this patch tested?
Pass the CIs with the newly added test cases.
- `LiteralExpressionSuite` checks the `Literal.sql` round trip of `TIME(7)`,
`TIME(8)`, and `TIME(9)` literals with trailing fractional zeros, a whole
second, and no trailing zeros.
- `ParametersSuite` checks the comparison and filter results above with
named and positional arguments and `EXECUTE IMMEDIATE`.
Without this fix, they fail like the following.
```
[info] - SPARK-60314: TimeType(7-9) sql keeps the precision on a round trip
*** FAILED ***
[info] TimeType(6) did not equal TimeType(7)
(LiteralExpressionSuite.scala:729)
```
```
[info] - SPARK-60314: TIME(7)-TIME(9) literal argument keeps its precision
*** FAILED ***
[info] TimeType(6) did not equal TimeType(9) (ParametersSuite.scala:533)
```
The existing golden files are unchanged because all `TIME(7)`-`TIME(9)`
literals there have no trailing fractional zeros.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
