This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new a6207b99ee30 [SPARK-57808][SQL][DOC] Document the microsecond-only
limitation of typed encoders and UDFs for nanosecond timestamps
a6207b99ee30 is described below
commit a6207b99ee30c17c3f8f519b6bfd61f602415f2c
Author: AgenticSpark <[email protected]>
AuthorDate: Thu Jul 2 11:55:28 2026 +0200
[SPARK-57808][SQL][DOC] Document the microsecond-only limitation of typed
encoders and UDFs for nanosecond timestamps
### What changes were proposed in this pull request?
This adds a note to `docs/sql-ref-datatypes.md` under the
`TimestampNTZNanosType(precision)` / `TimestampLTZNanosType(precision)` entry.
The note documents that typed encoders and typed Scala/Java UDF boundaries use
the existing microsecond timestamp types for `java.time.Instant` and
`java.time.LocalDateTime`, while Kryo encoders bypass the SQL type system. This
is a sub-task of the SPARK-56822 nanosecond-timestamp SPIP.
### Why are the changes needed?
Users can otherwise hit silent microsecond truncation when working with
nanosecond timestamp values through typed encoders or typed UDFs. Per
SPARK-57808, the decision is to document this limitation rather than change the
typed encoder behavior. The note also points users to the schema-driven
workaround: declare `TimestampLTZNanosType` or `TimestampNTZNanosType` in the
schema and use the `Row` / `Encoders.row(schema)` path to preserve nanoseconds.
### Does this PR introduce _any_ user-facing change?
No. Documentation-only.
### How was this patch tested?
Docs-only change; no tests. Verified the referenced APIs, configuration,
and type names exist in the codebase: `ScalaReflection` maps `Instant` /
`LocalDateTime` to microsecond `TimestampType` / `TimestampNTZType`,
`Encoders.row(schema)` exists, and `spark.sql.timestampNanosTypes.enabled`,
`TimestampLTZNanosType`, and `TimestampNTZNanosType` exist.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI
Closes #56956 from AgenticSpark/agenticspark/SPARK-57808-doc-nano-encoders.
Authored-by: AgenticSpark <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit b3bc6fb46607534bef78f08abf32495bf48f22d9)
Signed-off-by: Max Gekk <[email protected]>
---
docs/sql-ref-datatypes.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/docs/sql-ref-datatypes.md b/docs/sql-ref-datatypes.md
index a5da7949d059..89fd79c48936 100644
--- a/docs/sql-ref-datatypes.md
+++ b/docs/sql-ref-datatypes.md
@@ -56,6 +56,7 @@ Spark SQL and DataFrames support the following data types:
hour, minute, and second. All operations are performed without taking any
time zone into account.
- Note: TIMESTAMP in Spark is a user-specified alias associated with one
of the TIMESTAMP_LTZ and TIMESTAMP_NTZ variations. Users can set the default
timestamp type as `TIMESTAMP_LTZ`(default value) or `TIMESTAMP_NTZ` via the
configuration `spark.sql.timestampType`.
- `TimestampNTZNanosType(precision)` / `TimestampLTZNanosType(precision)`:
Preview nanosecond-capable variants of `TIMESTAMP_NTZ` and `TIMESTAMP_LTZ` with
fractional seconds precision `precision` in `[7, 9]`. Unparameterized
`TIMESTAMP`, `TIMESTAMP_NTZ`, and `TIMESTAMP_LTZ` remain microsecond types. In
schema-driven Dataset/DataFrame conversion, Spark maps `TimestampNTZNanosType`
to `java.time.LocalDateTime` and `TimestampLTZNanosType` to
`java.time.Instant`; values with more sub-micro [...]
+ - Note: Nanosecond precision is only available through the schema-driven
APIs above. Typed encoders derived from
`java.time.Instant`/`java.time.LocalDateTime` (for example `ds.as[T]` for a
case class and `spark.createDataFrame(Seq(...))` without an explicit schema)
bind these types to the microsecond `TIMESTAMP_LTZ`/`TIMESTAMP_NTZ` types, and
typed Scala/Java UDFs such as `udf((i: java.time.Instant) => ...)` coerce their
inputs to microsecond precision at the UDF boundary. Kryo encod [...]
* Interval types
- `YearMonthIntervalType(startField, endField)`: Represents a year-month
interval which is made up of a contiguous subset of the following fields:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
