stevomitric commented on code in PR #59579:
URL: https://github.com/apache/spark/pull/59579#discussion_r4323306334
##########
docs/sql-ref-datatypes.md:
##########
@@ -57,6 +57,7 @@ Spark SQL and DataFrames support the following data types:
- Note: TIMESTAMP in Spark is a user-specified alias associated with one
of the TIMESTAMP_LTZ and TIMESTAMP_NTZ variations. Users can set the default
timestamp type as `TIMESTAMP_LTZ`(default value) or `TIMESTAMP_NTZ` via the
configuration `spark.sql.timestampType`.
- `TimestampNTZNanosType(precision)` / `TimestampLTZNanosType(precision)`:
Preview nanosecond-capable variants of `TIMESTAMP_NTZ` and `TIMESTAMP_LTZ` with
fractional seconds precision `precision` in `[7, 9]`. Unparameterized
`TIMESTAMP`, `TIMESTAMP_NTZ`, and `TIMESTAMP_LTZ` remain microsecond types. In
schema-driven Dataset/DataFrame conversion, Spark maps `TimestampNTZNanosType`
to `java.time.LocalDateTime` and `TimestampLTZNanosType` to
`java.time.Instant`; values with more sub-micro digits than declared by
`precision` are floor-truncated to that precision. Enable the preview feature
with `SET spark.sql.timestampNanosTypes.enabled=true;` before using these types
in schemas or SQL.
- Note: Nanosecond precision is only available through the schema-driven
APIs above. Typed encoders derived from
`java.time.Instant`/`java.time.LocalDateTime` (for example `ds.as[T]` for a
case class and `spark.createDataFrame(Seq(...))` without an explicit schema)
bind these types to the microsecond `TIMESTAMP_LTZ`/`TIMESTAMP_NTZ` types, and
typed Scala/Java UDFs such as `udf((i: java.time.Instant) => ...)` coerce their
inputs to microsecond precision at the UDF boundary. Kryo encoders bypass the
SQL type system entirely and likewise do not preserve nanoseconds. To retain
nanoseconds, use a schema that declares
`TimestampLTZNanosType`/`TimestampNTZNanosType` together with the schema-driven
`Row`/`Encoders.row(schema)` path.
+ - Note: The range is the same as for the microsecond types (epoch
microseconds in a 64-bit long, about years -290308 to +294247), so years after
9999 are valid values. Adding an interval (`+ INTERVAL`, `timestampadd`) past
this range raises `DATETIME_OVERFLOW`, and casting an out-of-range string
returns `NULL`, or raises `CAST_INVALID_INPUT` under ANSI mode. Parquet
`TIMESTAMP(NANOS)` and Avro `timestamp-nanos`/`local-timestamp-nanos` store a
64-bit count of epoch nanoseconds, so they only hold `1677-09-21
00:12:43.145224192` to `2262-04-11 23:47:16.854775807` UTC. Arrow
`timestamp[ns]`, as used by pandas conversion and Spark Connect, starts one
nanosecond later, at `1677-09-21 00:12:43.145224193` UTC, because pandas
reserves the 64-bit minimum for `NaT`. Writing a value outside these ranges
raises `DATETIME_OVERFLOW`.
Review Comment:
Good point, we initially wanted the layout to be same as micros so same
limits apply, but lets keep it sql-compliant for now (easier to extend it later
then to shrink it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
