uros-b commented on code in PR #59575:
URL: https://github.com/apache/spark/pull/59575#discussion_r4313254882
##########
docs/sql-ref-datatypes.md:
##########
@@ -56,7 +56,7 @@ Spark SQL and DataFrames support the following data types:
hour, minute, and second. All operations are performed without taking any
time zone into account.
- Note: TIMESTAMP in Spark is a user-specified alias associated with one
of the TIMESTAMP_LTZ and TIMESTAMP_NTZ variations. Users can set the default
timestamp type as `TIMESTAMP_LTZ`(default value) or `TIMESTAMP_NTZ` via the
configuration `spark.sql.timestampType`.
- `TimestampNTZNanosType(precision)` / `TimestampLTZNanosType(precision)`:
Preview nanosecond-capable variants of `TIMESTAMP_NTZ` and `TIMESTAMP_LTZ` with
fractional seconds precision `precision` in `[7, 9]`. Unparameterized
`TIMESTAMP`, `TIMESTAMP_NTZ`, and `TIMESTAMP_LTZ` remain microsecond types. In
schema-driven Dataset/DataFrame conversion, Spark maps `TimestampNTZNanosType`
to `java.time.LocalDateTime` and `TimestampLTZNanosType` to
`java.time.Instant`; values with more sub-micro digits than declared by
`precision` are floor-truncated to that precision. Enable the preview feature
with `SET spark.sql.timestampNanosTypes.enabled=true;` before using these types
in schemas or SQL.
- - Note: Nanosecond precision is only available through the schema-driven
APIs above. Typed encoders derived from
`java.time.Instant`/`java.time.LocalDateTime` (for example `ds.as[T]` for a
case class and `spark.createDataFrame(Seq(...))` without an explicit schema)
bind these types to the microsecond `TIMESTAMP_LTZ`/`TIMESTAMP_NTZ` types, and
typed Scala/Java UDFs such as `udf((i: java.time.Instant) => ...)` coerce their
inputs to microsecond precision at the UDF boundary. Kryo encoders bypass the
SQL type system entirely and likewise do not preserve nanoseconds. To retain
nanoseconds, use a schema that declares
`TimestampLTZNanosType`/`TimestampNTZNanosType` together with the schema-driven
`Row`/`Encoders.row(schema)` path.
+ - Note: Nanosecond precision is only available through the schema-driven
APIs above. Typed encoders derived from
`java.time.Instant`/`java.time.LocalDateTime` (for example `ds.as[T]` for a
case class and `spark.createDataFrame(Seq(...))` without an explicit schema)
bind these types to the microsecond `TIMESTAMP_LTZ`/`TIMESTAMP_NTZ` types, and
typed Scala/Java UDFs such as `udf((i: java.time.Instant) => ...)` take
microsecond inputs, so a nanosecond argument needs an explicit `CAST` to the
microsecond type. Kryo encoders bypass the SQL type system entirely and
likewise do not preserve nanoseconds. To retain nanoseconds, use a schema that
declares `TimestampLTZNanosType`/`TimestampNTZNanosType` together with the
schema-driven `Row`/`Encoders.row(schema)` path.
Review Comment:
The nanos type note still says typed UDFs coerce nanos inputs to
microseconds. After this PR, they reject them in both modes.
The note (SPARK-57808) says typed Scala/Java UDFs such as udf((i:
java.time.Instant) => ...) "coerce their inputs to microsecond precision at the
UDF boundary".
The PR body says a non-ANSI Scala UDF typed as a microsecond timestamp "now
rejects a nanos argument, as ANSI already did". After this PR the note is wrong
in both modes; for ANSI it is already wrong on master. Change it to say that an
explicit CAST is required.
No test covers this user-facing change, and the error class probably differs
from the built-in case (conf med):
- ScalaUDF has no checkInputDataTypes. With no cast inserted, the failure
should come from encoder binding: ResolveEncodersInUDF resolves UpCast(nanos,
TIMESTAMP) and fails canUpCast.
- That is an up-cast error, not the UNEXPECTED_INPUT_TYPE shown for
built-ins. Procedures do report UNEXPECTED_INPUT_TYPE, via Call.checkArgTypes.
Add a typed-UDF test and a procedure test that pin these errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
