This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 6b6d13415abd [SPARK-57285][SQL] Route nanosecond timestamp
cast-to-string through the Types Framework
6b6d13415abd is described below
commit 6b6d13415abdd6ebea054ae9d3b4cc496e7eaf0e
Author: Maxim Gekk <[email protected]>
AuthorDate: Thu Jun 11 00:02:40 2026 +0200
[SPARK-57285][SQL] Route nanosecond timestamp cast-to-string through the
Types Framework
### What changes were proposed in this pull request?
This PR makes the Types Framework (`TypeApiOps`) the single integration
point for nanosecond timestamp `CAST(... AS STRING)`, for both the interpreted
and codegen paths.
Specifically:
- `TypeApiOps.apply` gains a by-name `zoneId` parameter that defaults to
the session-local time zone config (`SqlApiConf.get.sessionLocalTimeZone`) and
is threaded into the `TIMESTAMP_LTZ` nanos ops. It is by-name so the zone is
forced only when the LTZ ops is actually constructed: zone-independent
(`TimeType`, `TIMESTAMP_NTZ` nanos) and unsupported types never evaluate it,
which matters because a `CAST`'s zone is unresolved (`None.get`) until a time
zone is assigned.
- `TimestampLTZNanosTypeApiOps` now carries its `ZoneId` as a required
constructor parameter and holds the fraction formatter in a `transient private
lazy val`, so the formatter is built once per ops instance (once per cast, per
task) rather than per row. `TimestampNTZNanosTypeApiOps` stays zone-independent
(UTC).
- `ToStringBase` no longer bypasses or special-cases the framework: both
the interpreted and codegen paths dispatch uniformly through `TypeApiOps(from,
zoneId)`. `CAST` passes its resolved zone; zone-less callers accept the
session-zone default.
- The nanosecond ops implement `formatExternal` to render the external Row
value (`Instant` for LTZ, `LocalDateTime` for NTZ) at the column precision, so
`Row.json` / `Row.prettyJson` render nanos columns through the `formatExternal`
routing introduced in #56392 (SPARK-57338). The two-arg `formatExternal(value,
nested)` returns `None`, leaving the `HiveResult` output path unchanged.
- The now-unused `UNSUPPORTED_FEATURE.TIMESTAMP_NANOS_TO_STRING` error
condition and its `DataTypeErrors` helper are removed.
The microsecond timestamp types (`TIMESTAMP` / `TIMESTAMP_NTZ`) remain
handled inline in `ToStringBase` and are out of scope.
### Why are the changes needed?
SPARK-57256 implemented nanosecond cast-to-string inline in `ToStringBase`,
deliberately bypassing the framework because the zone-less
`TypeApiOps.format(v)` cannot render LTZ in the session time zone. That left
nanos cast-to-string as a one-off integration outside the framework,
inconsistent with the SPIP direction (SPARK-56822) of wiring the new types
through the centralized `TypeOps` / `TypeApiOps`. This PR closes that gap.
### Does this PR introduce _any_ user-facing change?
Yes, two narrow deltas:
1. `CAST(... AS STRING)` on nanosecond `TIMESTAMP_LTZ` / `TIMESTAMP_NTZ`
now dispatches through the Types Framework. The rendered string is unchanged
(zone-aware LTZ, zone-independent NTZ, precision flooring, trailing-zero
trimming); the only behavioral change is that the zone-less type-level
`format()` no longer throws `UNSUPPORTED_FEATURE.TIMESTAMP_NANOS_TO_STRING`
(that error condition is removed).
2. `Row.json` / `Row.prettyJson` on a nanosecond `TIMESTAMP_LTZ` /
`TIMESTAMP_NTZ` column now renders the value (NTZ zone-independent, LTZ in
`spark.sql.session.timeZone`) via the `formatExternal` routing added in #56392
(SPARK-57338), instead of raising
`UNSUPPORTED_FEATURE.TIMESTAMP_NANOS_TO_STRING`.
`EXPLAIN` and SQL-literal `toSQLValue` are unchanged: they don't route
through the framework (`Literal.toString` / `Literal.sql` render via
`value.toString`, and `toSQLValue` has no production caller).
### How was this patch tested?
- Updated `TimestampNanosTypeOpsSuite` to cover the new behavior: NTZ
renders zone-independently, LTZ renders in the zone carried by the ops
instance, and zone-less LTZ now renders in the session-local time zone (instead
of raising), exercising precision flooring.
- Added a `RowJsonSuite` test that `Row.json` renders a nanosecond
`TIMESTAMP_NTZ` / `TIMESTAMP_LTZ` column at the column precision (this PR is
rebased on the merged #56392, which routes `Row.jsonValue` through
`formatExternal`).
- `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `ToPrettyStringSuite`, and
`TimestampNanosRowSuite` stay green (281 tests pass), and `SparkThrowableSuite`
(33 tests) confirms the removed error condition leaves the error framework
consistent.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56355 from MaxGekk/nanos-typeframework-formatter.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit 1175d40c3b306d3626c0dfa8d25a8aef035113a7)
Signed-off-by: Max Gekk <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 5 --
.../apache/spark/sql/errors/DataTypeErrors.scala | 6 --
.../sql/types/ops/TimestampNanosTypeApiOps.scala | 85 +++++++++++++------
.../apache/spark/sql/types/ops/TypeApiOps.scala | 26 ++++--
.../sql/catalyst/expressions/ToStringBase.scala | 49 +++++------
.../catalyst/types/ops/TimestampNanosTypeOps.scala | 7 +-
.../scala/org/apache/spark/sql/RowJsonSuite.scala | 31 ++++---
.../types/ops/TimestampNanosTypeOpsSuite.scala | 96 +++++++++++++++++++---
8 files changed, 202 insertions(+), 103 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index bf5b3092511f..f4b6b8980059 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -8378,11 +8378,6 @@
"Temporary views cannot be created with the WITH SCHEMA clause.
Recreate the temporary view when the underlying schema changes, or use a
persisted view."
]
},
- "TIMESTAMP_NANOS_TO_STRING" : {
- "message" : [
- "Converting values of the nanosecond timestamp type <dataType> to a
string."
- ]
- },
"TIMESTAMP_NANOS_WITH_LEGACY_TIME_PARSER" : {
"message" : [
"Parsing or formatting nanosecond-precision timestamps
(TIMESTAMP_LTZ/TIMESTAMP_NTZ with precision in [7, 9]) under the LEGACY time
parser policy. Set <config> to CORRECTED."
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala
b/sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala
index 7a4b2a56d25e..955e242d4ab5 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala
@@ -299,10 +299,4 @@ private[sql] object DataTypeErrors extends
DataTypeErrorsBase {
"configValue" -> "true"),
cause = null)
}
-
- def cannotConvertNanosTimestampToStringError(dataType: DataType): Throwable
= {
- new SparkUnsupportedOperationException(
- errorClass = "UNSUPPORTED_FEATURE.TIMESTAMP_NANOS_TO_STRING",
- messageParameters = Map("dataType" -> toSQLType(dataType)))
- }
}
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
b/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
index fc3bb947ee8f..d9b3a431f1ec 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
@@ -17,24 +17,30 @@
package org.apache.spark.sql.types.ops
+import java.time.{Instant, LocalDateTime, ZoneId, ZoneOffset}
+
import org.apache.spark.sql.catalyst.encoders.AgnosticEncoder
import
org.apache.spark.sql.catalyst.encoders.AgnosticEncoders.{InstantNanosEncoder,
LocalDateTimeNanosEncoder}
+import org.apache.spark.sql.catalyst.util.TimestampFormatter
import org.apache.spark.sql.errors.{DataTypeErrors, DataTypeErrorsBase}
import org.apache.spark.sql.types.{TimestampLTZNanosType,
TimestampNTZNanosType}
+import org.apache.spark.unsafe.types.TimestampNanosVal
/**
* Client-side (spark-api) operations shared by the nanosecond timestamp types
* (TimestampNTZNanosType and TimestampLTZNanosType).
*
* Internal values are [[org.apache.spark.unsafe.types.TimestampNanosVal]]
(epoch micros + nanos
- * within the micro). The two concrete subclasses differ only in their
DataType and SQL-literal
- * prefix; storage and formatting are identical.
+ * within the micro). The two concrete subclasses differ only in their
DataType, SQL-literal
+ * prefix, and time-zone handling; storage is identical.
*
- * SCOPE (SPARK-57207): this issue wires physical representation, literals,
row accessors, and
- * codegen class selection. CAST to STRING is implemented separately,
zone-aware, in ToStringBase
- * (SPARK-57256). The zone-less, type-level format() here (and the
toSQLValue() that delegates to
- * it) still raises the user-facing
UNSUPPORTED_FEATURE.TIMESTAMP_NANOS_TO_STRING error, since LTZ
- * rendering needs the session time zone that this op does not have.
+ * CAST to STRING (SPARK-57285): the Types Framework is the single integration
point for
+ * nanosecond timestamp cast-to-string, via the zone-less [[format]] /
[[formatUTF8]]. NTZ
+ * rendering is zone-independent (the value is the UTC-grid wall clock). LTZ
rendering depends on
+ * the session time zone, so the LTZ ops carries a ZoneId and builds its
formatter once per
+ * instance. The zone is threaded in via TypeApiOps.apply(dt, zoneId): CAST
passes the cast's
+ * resolved session zone, while zone-less callers (Row JSON via
formatExternal) accept the
+ * default, the session-local time zone config.
*
* Dataset encoders are wired here to the precision-aware leaves added by
SPARK-57033
* (LocalDateTimeNanosEncoder / InstantNanosEncoder), so that turning on the
Types Framework
@@ -47,27 +53,20 @@ abstract class TimestampNanosTypeApiOps extends TypeApiOps
with DataTypeErrorsBa
/** SQL literal prefix for this type, e.g. "TIMESTAMP_NTZ" or
"TIMESTAMP_LTZ". */
protected def sqlTypeName: String
- // ==================== String Formatting ====================
+ /** The fractional-second precision in [7, 9] of this type. */
+ protected def precision: Int
- // CAST to STRING for the nanosecond timestamp types is handled zone-aware
by ToStringBase
- // (SPARK-57256), alongside the microsecond timestamp types, because LTZ
rendering depends on the
- // session time zone that this zone-less, type-level formatter does not
have. The remaining
- // zone-less callers (EXPLAIN plan output and SQL-literal rendering via
toSQLValue) still raise a
- // user-facing unsupported-feature error here rather than silently
truncating to microseconds.
- override def format(v: Any): String =
- throw DataTypeErrors.cannotConvertNanosTimestampToStringError(dataType)
+ // ==================== String Formatting ====================
// Row JSON (Row.json / Row.prettyJson) holds the external Row value
(java.time.Instant for LTZ,
- // java.time.LocalDateTime for NTZ), but rendering nanosecond timestamps
from this zone-less path
- // is not supported yet (same reason as format above), so raise the
user-facing
- // UNSUPPORTED_FEATURE.TIMESTAMP_NANOS_TO_STRING error here rather than
silently truncating.
- override def formatExternal(value: Any): Option[String] =
- throw DataTypeErrors.cannotConvertNanosTimestampToStringError(dataType)
-
- // The Hive result path (HiveResult.toHiveString) is zone-aware and renders
nanosecond timestamps
- // through its own default formatter, so return None here to fall through to
it rather than raise
- // the unsupported-rendering error that the zone-less single-arg
formatExternal above throws. This
- // is a temporary override until nanos external rendering is unified across
the two paths.
+ // java.time.LocalDateTime for NTZ); each subclass overrides the single-arg
formatExternal to
+ // render it through the same formatter as its zone-aware cast-to-string, so
Row JSON shows the
+ // nanosecond value rather than silently truncating to microseconds via the
legacy path.
+
+ // The Hive result path (HiveResult.toHiveString) renders nanosecond
timestamps through its own
+ // zone-aware default formatter, so return None here to fall through to it
rather than to the
+ // subclass single-arg rendering. This is a temporary split until nanos
external rendering is
+ // unified across the zone-less (Row JSON) and zone-aware (Hive) paths.
override def formatExternal(value: Any, nested: Boolean): Option[String] =
None
override def toSQLValue(v: Any): String = s"$sqlTypeName '${format(v)}'"
@@ -96,6 +95,20 @@ abstract class TimestampNanosTypeApiOps extends TypeApiOps
with DataTypeErrorsBa
class TimestampNTZNanosTypeApiOps(val t: TimestampNTZNanosType) extends
TimestampNanosTypeApiOps {
override def dataType: TimestampNTZNanosType = t
override protected def sqlTypeName: String = "TIMESTAMP_NTZ"
+ override protected def precision: Int = t.precision
+
+ // NTZ rendering is zone-independent: the value is the UTC-grid wall clock,
so a UTC formatter
+ // produces the same string regardless of the session zone.
+ @transient private lazy val formatter =
TimestampFormatter.getFractionFormatter(ZoneOffset.UTC)
+
+ override def format(v: Any): String =
+ formatter.formatWithoutTimeZoneNanos(v.asInstanceOf[TimestampNanosVal],
precision)
+
+ // Row JSON holds the external java.time.LocalDateTime
(LocalDateTimeNanosEncoder, SPARK-57033)
+ // already at the column precision; render it zone-independently through the
same UTC formatter
+ // as the internal-value format above.
+ override def formatExternal(value: Any): Option[String] =
+ Some(formatter.format(value.asInstanceOf[LocalDateTime]))
// Mirrors RowEncoder.encoderForDataTypeDefault for TimestampNTZNanosType
(SPARK-57033):
// maps to java.time.LocalDateTime with the column precision.
@@ -107,11 +120,31 @@ class TimestampNTZNanosTypeApiOps(val t:
TimestampNTZNanosType) extends Timestam
*
* @param t
* The TimestampLTZNanosType with precision information
+ * @param zoneId
+ * The time zone LTZ values are rendered in (LTZ is zone-aware).
`TypeApiOps.apply` threads in
+ * the session zone: the cast's resolved zone for CAST, or the session-local
time zone config
+ * for zone-less render callers (Row JSON via formatExternal).
* @since 4.3.0
*/
-class TimestampLTZNanosTypeApiOps(val t: TimestampLTZNanosType) extends
TimestampNanosTypeApiOps {
+class TimestampLTZNanosTypeApiOps(val t: TimestampLTZNanosType, zoneId: ZoneId)
+ extends TimestampNanosTypeApiOps {
override def dataType: TimestampLTZNanosType = t
override protected def sqlTypeName: String = "TIMESTAMP_LTZ"
+ override protected def precision: Int = t.precision
+
+ // LTZ rendering depends on the session time zone, carried by this ops
instance (the cast's
+ // resolved zone, or the session-local time zone config for the zone-less
framework lookup). The
+ // fraction formatter is built once here (per cast / per ops) rather than
per row.
+ @transient private lazy val formatter =
TimestampFormatter.getFractionFormatter(zoneId)
+
+ override def format(v: Any): String =
+ formatter.formatNanos(v.asInstanceOf[TimestampNanosVal], precision)
+
+ // Row JSON holds the external java.time.Instant (InstantNanosEncoder,
SPARK-57033) already at the
+ // column precision; render it in the ops' session zone through the same
formatter as the
+ // internal-value format above.
+ override def formatExternal(value: Any): Option[String] =
+ Some(formatter.format(value.asInstanceOf[Instant]))
// Mirrors RowEncoder.encoderForDataTypeDefault for TimestampLTZNanosType
(SPARK-57033):
// maps to java.time.Instant with the column precision.
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
b/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
index 40a158d70af3..d7575976e1f5 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
@@ -17,9 +17,12 @@
package org.apache.spark.sql.types.ops
+import java.time.ZoneId
+
import org.apache.arrow.vector.types.pojo.ArrowType
import org.apache.spark.sql.catalyst.encoders.AgnosticEncoder
+import org.apache.spark.sql.catalyst.util.SparkDateTimeUtils
import org.apache.spark.sql.internal.SqlApiConf
import org.apache.spark.sql.types.{DataType, TimestampLTZNanosType,
TimestampNTZNanosType, TimeType}
import org.apache.spark.unsafe.types.UTF8String
@@ -127,11 +130,11 @@ trait TypeApiOps extends Serializable {
* for TimeType).
*
* Consumer: Row JSON (Row.json / Row.prettyJson). Semantics:
- * - Some(s): s is used as the rendered JSON string.
+ * - Some(s): s is used as the rendered JSON string (e.g. the nanosecond
timestamp types
+ * render the external Instant/LocalDateTime at the column precision).
* - None: Row JSON falls back to its legacy toJsonDefault rendering.
* - throw: an implementation may raise instead, to signal that rendering
this type on this
- * zone-less path is unsupported (e.g. the nanosecond timestamp types
raise
- * UNSUPPORTED_FEATURE.TIMESTAMP_NANOS_TO_STRING).
+ * zone-less path is unsupported.
*
* Returning None silently routes the type into the legacy path, so a type
that must be rendered
* by the framework should override this (every currently registered type
does).
@@ -144,8 +147,8 @@ trait TypeApiOps extends Serializable {
* Semantics mirror the single-arg overload: Some(s) is used directly, None
falls back to
* HiveResult's zone-aware legacy rendering. The default delegates to the
single-arg overload;
* override it separately when the two consumers need different behavior
(e.g. the nanosecond
- * timestamp types throw on the zone-less Row JSON path but return None here
so the zone-aware
- * Hive path renders them).
+ * timestamp types render the external value on the zone-less Row JSON path
but return None here
+ * so the zone-aware Hive path renders them through its own formatter).
*/
def formatExternal(value: Any, nested: Boolean): Option[String] =
formatExternal(value)
@@ -172,15 +175,24 @@ object TypeApiOps {
*
* @param dt
* the DataType to get operations for
+ * @param zoneId
+ * the session time zone for zone-aware rendering (TIMESTAMP_LTZ nanos).
CAST passes the
+ * cast's resolved zone; zone-less callers (Row JSON via formatExternal)
accept the default,
+ * the session-local time zone config. By-name so it is forced only when
LTZ is constructed:
+ * zone-independent and unsupported types never evaluate it, which matters
because a CAST's
+ * zone is unresolved (`None.get`) until the time zone is assigned.
* @return
* Some(TypeApiOps) if supported, None otherwise
*/
- def apply(dt: DataType): Option[TypeApiOps] = {
+ def apply(
+ dt: DataType,
+ zoneId: => ZoneId =
SparkDateTimeUtils.getZoneId(SqlApiConf.get.sessionLocalTimeZone))
+ : Option[TypeApiOps] = {
if (!SqlApiConf.get.typesFrameworkEnabled) return None
dt match {
case tt: TimeType => Some(new TimeTypeApiOps(tt))
case t: TimestampNTZNanosType => Some(new TimestampNTZNanosTypeApiOps(t))
- case t: TimestampLTZNanosType => Some(new TimestampLTZNanosTypeApiOps(t))
+ case t: TimestampLTZNanosType => Some(new TimestampLTZNanosTypeApiOps(t,
zoneId))
// Add new types here - single registration point
case _ => None
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
index 561fd33b0a18..3a6ae2c2045f 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
@@ -29,7 +29,7 @@ import org.apache.spark.sql.internal.SQLConf.BinaryOutputStyle
import org.apache.spark.sql.types._
import org.apache.spark.sql.types.ops.TypeApiOps
import org.apache.spark.unsafe.UTF8StringBuilder
-import org.apache.spark.unsafe.types.{CalendarInterval, TimestampNanosVal,
UTF8String}
+import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
import org.apache.spark.util.ArrayImplicits._
import org.apache.spark.util.SparkStringUtils
@@ -66,16 +66,14 @@ trait ToStringBase { self: UnaryExpression with
TimeZoneAwareExpression =>
case NoConstraint => castToString(from)
}
- private def castToString(from: DataType): Any => UTF8String = from match {
- // Nanosecond timestamp string formatting is zone-aware (LTZ renders in
the session time zone),
- // so it lives in castToStringDefault alongside the microsecond timestamp
types rather than the
- // zone-less Types Framework formatter (SPARK-57256).
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType =>
castToStringDefault(from)
- case _ =>
- TypeApiOps(from)
- .map(ops => acceptAny[Any](v => ops.formatUTF8(v)))
- .getOrElse(castToStringDefault(from))
- }
+ // The Types Framework is the single integration point for framework types'
cast-to-string, via
+ // the zone-less formatUTF8. The cast's session zone is threaded into the
lookup so TIMESTAMP_LTZ
+ // nanos renders in it; zone-independent types (TimeType, TIMESTAMP_NTZ
nanos) ignore it
+ // (SPARK-57285).
+ private def castToString(from: DataType): Any => UTF8String =
+ TypeApiOps(from, zoneId)
+ .map(ops => acceptAny[Any](ops.formatUTF8))
+ .getOrElse(castToStringDefault(from))
private def castToStringDefault(from: DataType): Any => UTF8String = from
match {
case CalendarIntervalType =>
@@ -87,12 +85,6 @@ trait ToStringBase { self: UnaryExpression with
TimeZoneAwareExpression =>
acceptAny[Long](t => UTF8String.fromString(timestampFormatter.format(t)))
case TimestampNTZType =>
acceptAny[Long](t =>
UTF8String.fromString(timestampNTZFormatter.format(t)))
- case t: TimestampLTZNanosType =>
- acceptAny[TimestampNanosVal](v =>
- UTF8String.fromString(timestampFormatter.formatNanos(v, t.precision)))
- case t: TimestampNTZNanosType =>
- acceptAny[TimestampNanosVal](v =>
-
UTF8String.fromString(timestampNTZFormatter.formatWithoutTimeZoneNanos(v,
t.precision)))
case _: TimeType =>
acceptAny[Long](t => UTF8String.fromString(timeFormatter.format(t)))
case ArrayType(et, _) =>
@@ -246,18 +238,17 @@ trait ToStringBase { self: UnaryExpression with
TimeZoneAwareExpression =>
ctx.addReferenceObj("timestampNTZFormatter", timestampNTZFormatter),
timestampNTZFormatter.getClass)
(c, evPrim) => code"$evPrim = UTF8String.fromString($tf.format($c));"
- case t: TimestampLTZNanosType =>
- val tf = JavaCode.global(
- ctx.addReferenceObj("timestampFormatter", timestampFormatter),
- timestampFormatter.getClass)
- (c, evPrim) =>
- code"$evPrim = UTF8String.fromString($tf.formatNanos($c,
${t.precision}));"
- case t: TimestampNTZNanosType =>
- val tf = JavaCode.global(
- ctx.addReferenceObj("timestampNTZFormatter", timestampNTZFormatter),
- timestampNTZFormatter.getClass)
- (c, evPrim) =>
- code"$evPrim =
UTF8String.fromString($tf.formatWithoutTimeZoneNanos($c, ${t.precision}));"
+ case _: TimestampNTZNanosType | _: TimestampLTZNanosType =>
+ // Route nanosecond timestamp cast-to-string through the Types
Framework: emit a runtime
+ // call into the ops reference object. The cast's session zone is
threaded into the lookup
+ // so LTZ carries it; NTZ is zone-independent (SPARK-57285).
+ val ops = TypeApiOps(from, zoneId).get
+ // Pin the reference-object cast type to the public TypeApiOps class;
the runtime ops class
+ // lives in sql/api, so the inferred concrete-class cast would be
unnecessarily specific.
+ val opsRef = JavaCode.global(
+ ctx.addReferenceObj("typeApiOps", ops, classOf[TypeApiOps].getName),
+ classOf[TypeApiOps])
+ (c, evPrim) => code"$evPrim = $opsRef.formatUTF8($c);"
case _: TimeType =>
val tf = JavaCode.global(
ctx.addReferenceObj("timeFormatter", timeFormatter),
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOps.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOps.scala
index 9957367ed039..96fe32999b93 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOps.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOps.scala
@@ -17,7 +17,7 @@
package org.apache.spark.sql.catalyst.types.ops
-import java.time.{Instant, LocalDateTime}
+import java.time.{Instant, LocalDateTime, ZoneOffset}
import org.apache.spark.SparkIllegalArgumentException
import org.apache.spark.sql.catalyst.InternalRow
@@ -130,8 +130,11 @@ case class TimestampNTZNanosTypeOps(override val t:
TimestampNTZNanosType)
* The TimestampLTZNanosType with precision information
* @since 4.3.0
*/
+// Server-side TypeOps is used only for physical type, literals, row
accessors, and serde - never
+// for rendering (cast-to-string flows through TypeApiOps.apply). So the
rendering zone is unused
+// here; pass UTC rather than reading the session config on every construction.
case class TimestampLTZNanosTypeOps(override val t: TimestampLTZNanosType)
- extends TimestampLTZNanosTypeApiOps(t) with TimestampNanosTypeOps {
+ extends TimestampLTZNanosTypeApiOps(t, ZoneOffset.UTC) with
TimestampNanosTypeOps {
override def getPhysicalType: PhysicalDataType =
PhysicalTimestampLTZNanosType
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/RowJsonSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/RowJsonSuite.scala
index edc7b9512d38..183eefaa8d75 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/RowJsonSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/RowJsonSuite.scala
@@ -21,7 +21,7 @@ import java.time.{LocalDate, LocalDateTime, LocalTime,
ZoneOffset}
import org.json4s.JsonAST.{JArray, JBool, JDecimal, JDouble, JLong, JNull,
JObject, JString, JValue}
-import org.apache.spark.{SparkFunSuite, SparkIllegalArgumentException,
SparkUnsupportedOperationException}
+import org.apache.spark.{SparkFunSuite, SparkIllegalArgumentException}
import org.apache.spark.sql.catalyst.encoders.{ExamplePoint, ExamplePointUDT}
import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
import org.apache.spark.sql.catalyst.plans.SQLHelper
@@ -174,24 +174,23 @@ class RowJsonSuite extends SparkFunSuite with SQLHelper {
}
}
- // Routing the external value through formatExternal must not silently
change the nanosecond
- // timestamp behavior: those ops raise the clean unsupported-rendering error
directly from
- // formatExternal instead of mis-rendering the external value as a
microsecond timestamp.
- test("SPARK-57338: nanosecond timestamp column raises the
unsupported-rendering error in JSON") {
+ // With nanosecond cast-to-string now supported through the framework
(SPARK-57285), Row JSON
+ // routes the external Row value (LocalDateTime for NTZ, Instant for LTZ)
through formatExternal
+ // and renders it at the column precision, rather than raising an
unsupported-rendering error.
+ test("SPARK-57338: nanosecond timestamp column renders the external value in
JSON") {
+ def nanosRowJson(value: Any, dt: DataType): JValue =
+ new GenericRowWithSchema(Array(value), new StructType().add("a",
dt)).jsonValue
withSQLConf(
SQLConf.TYPES_FRAMEWORK_ENABLED.key -> "true",
- SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true",
+ SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
val ldt = LocalDateTime.of(2020, 1, 1, 12, 0, 0, 123456789)
- Seq[(Any, DataType)](
- ldt -> TimestampNTZNanosType(9),
- ldt.toInstant(ZoneOffset.UTC) -> TimestampLTZNanosType(9)).foreach {
case (value, dt) =>
- checkError(
- exception = intercept[SparkUnsupportedOperationException] {
- new GenericRowWithSchema(Array(value), new StructType().add("a",
dt)).jsonValue
- },
- condition = "UNSUPPORTED_FEATURE.TIMESTAMP_NANOS_TO_STRING",
- parameters = Map("dataType" -> toSQLType(dt)))
- }
+ // NTZ renders zone-independently (the value is the UTC-grid wall clock).
+ assert(nanosRowJson(ldt, TimestampNTZNanosType(9)) ===
+ JObject("a" -> JString("2020-01-01 12:00:00.123456789")))
+ // LTZ renders in the session zone; with UTC it matches the NTZ wall
clock.
+ assert(nanosRowJson(ldt.toInstant(ZoneOffset.UTC),
TimestampLTZNanosType(9)) ===
+ JObject("a" -> JString("2020-01-01 12:00:00.123456789")))
}
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOpsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOpsSuite.scala
index 2d69729540cd..b31fe310755f 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOpsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOpsSuite.scala
@@ -17,9 +17,9 @@
package org.apache.spark.sql.catalyst.types.ops
-import java.time.{Instant, LocalDateTime}
+import java.time.{Instant, LocalDateTime, ZoneOffset}
-import org.apache.spark.{SparkFunSuite, SparkUnsupportedOperationException}
+import org.apache.spark.SparkFunSuite
import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
import org.apache.spark.sql.catalyst.encoders.{AgnosticEncoder,
ExpressionEncoder}
import
org.apache.spark.sql.catalyst.encoders.AgnosticEncoders.{InstantNanosEncoder,
LocalDateTimeNanosEncoder, OptionEncoder}
@@ -30,8 +30,8 @@ import org.apache.spark.sql.catalyst.types.{PhysicalDataType,
PhysicalTimestampL
import org.apache.spark.sql.catalyst.util.DateTimeUtils
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{DataType, TimestampLTZNanosType,
TimestampNTZNanosType}
-import org.apache.spark.sql.types.ops.TypeApiOps
-import org.apache.spark.unsafe.types.TimestampNanosVal
+import org.apache.spark.sql.types.ops.{TimestampLTZNanosTypeApiOps, TypeApiOps}
+import org.apache.spark.unsafe.types.{TimestampNanosVal, UTF8String}
/**
* Tests for the Types Framework wiring of the nanosecond timestamp types
(SPARK-57207).
@@ -162,18 +162,90 @@ class TimestampNanosTypeOpsSuite extends SparkFunSuite
with SQLHelper {
}
}
- test("format and toSQLValue raise
UNSUPPORTED_FEATURE.TIMESTAMP_NANOS_TO_STRING") {
- allCases.foreach { case (dt, _, value) =>
- val ops = TypeApiOps(dt).get
- Seq[() => Any](() => ops.format(value), () =>
ops.toSQLValue(value)).foreach { call =>
- checkError(
- exception = intercept[SparkUnsupportedOperationException](call()),
- condition = "UNSUPPORTED_FEATURE.TIMESTAMP_NANOS_TO_STRING",
- parameters = Map("dataType" -> ("\"" + dt.sql + "\"")))
+ // SPARK-57285: a value with sub-precision (sub-nano) digits so flooring is
exercised; built from
+ // a UTC wall clock so the rendered NTZ/LTZ strings are predictable.
+ private val nanosLdt = LocalDateTime.of(2020, 1, 1, 0, 0, 0, 123456789)
+
+ // Expected fractional second after flooring to the given nanos precision
(trailing zeros are
+ // trimmed by the formatter, but 123456789 has none to trim at p in {7,8,9}).
+ private def expectedFraction(precision: Int): String = precision match {
+ case 7 => "1234567"
+ case 8 => "12345678"
+ case 9 => "123456789"
+ }
+
+ test("SPARK-57285: NTZ format renders zone-independently (no session zone
needed)") {
+ precisions.foreach { p =>
+ val ops = TypeApiOps(TimestampNTZNanosType(p)).get
+ val v = DateTimeUtils.localDateTimeToTimestampNanos(nanosLdt, p)
+ val expected = s"2020-01-01 00:00:00.${expectedFraction(p)}"
+ // NTZ is zone-independent: it renders the same regardless of the
session zone.
+ assert(ops.format(v) === expected, s"NTZ format for precision $p")
+ assert(ops.formatUTF8(v) === UTF8String.fromString(expected))
+ assert(ops.toSQLValue(v) === s"TIMESTAMP_NTZ '$expected'")
+ }
+ }
+
+ test("SPARK-57285: LTZ format renders in the zone carried by the ops
instance") {
+ precisions.foreach { p =>
+ val t = TimestampLTZNanosType(p)
+ // Physical value is the epoch instant of the UTC wall clock above.
+ val v =
DateTimeUtils.instantToTimestampNanos(nanosLdt.toInstant(ZoneOffset.UTC), p)
+ val frac = expectedFraction(p)
+ // UTC zone: the rendered wall clock matches the UTC instant.
+ val utcOps = new TimestampLTZNanosTypeApiOps(t, ZoneOffset.UTC)
+ assert(utcOps.format(v) === s"2020-01-01 00:00:00.$frac")
+ assert(utcOps.formatUTF8(v) === UTF8String.fromString(s"2020-01-01
00:00:00.$frac"))
+ assert(utcOps.toSQLValue(v) === s"TIMESTAMP_LTZ '2020-01-01
00:00:00.$frac'")
+ // A +05:00 zone shifts the rendered wall clock; the fraction is
unaffected.
+ val offsetOps = new TimestampLTZNanosTypeApiOps(t, ZoneOffset.ofHours(5))
+ assert(offsetOps.format(v) === s"2020-01-01 05:00:00.$frac")
+ }
+ }
+
+ test("SPARK-57285: LTZ zone-less lookup renders in the session-local time
zone config") {
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ precisions.foreach { p =>
+ val dt = TimestampLTZNanosType(p)
+ // TypeApiOps.apply has no cast zone, so it defaults to the
session-local time zone config.
+ val ops = TypeApiOps(dt).get
+ val v =
DateTimeUtils.instantToTimestampNanos(nanosLdt.toInstant(ZoneOffset.UTC), p)
+ val frac = expectedFraction(p)
+ assert(ops.format(v) === s"2020-01-01 00:00:00.$frac")
+ assert(ops.toSQLValue(v) === s"TIMESTAMP_LTZ '2020-01-01
00:00:00.$frac'")
}
}
}
+ test("SPARK-57285: NTZ formatExternal renders LocalDateTime at the column
precision") {
+ precisions.foreach { p =>
+ val ops = TypeApiOps(TimestampNTZNanosType(p)).get
+ // The encoder floors to column precision before handing the value to
Row JSON;
+ // round-trip through TimestampNanosVal to reproduce that flooring, then
format.
+ val v = DateTimeUtils.localDateTimeToTimestampNanos(nanosLdt, p)
+ val ldt = DateTimeUtils.timestampNanosToLocalDateTime(v)
+ val expected = s"2020-01-01 00:00:00.${expectedFraction(p)}"
+ assert(ops.formatExternal(ldt) === Some(expected), s"NTZ formatExternal
for precision $p")
+ }
+ }
+
+ test("SPARK-57285: LTZ formatExternal renders Instant at the column
precision in the ops zone") {
+ precisions.foreach { p =>
+ val t = TimestampLTZNanosType(p)
+ val v =
DateTimeUtils.instantToTimestampNanos(nanosLdt.toInstant(ZoneOffset.UTC), p)
+ val instant = DateTimeUtils.timestampNanosToInstant(v)
+ val frac = expectedFraction(p)
+ // UTC zone: rendered wall clock matches the UTC instant.
+ val utcOps = new TimestampLTZNanosTypeApiOps(t, ZoneOffset.UTC)
+ assert(utcOps.formatExternal(instant) === Some(s"2020-01-01
00:00:00.$frac"),
+ s"LTZ formatExternal for precision $p")
+ // A +05:00 zone shifts the rendered wall clock; the fraction is
unaffected.
+ val offsetOps = new TimestampLTZNanosTypeApiOps(t, ZoneOffset.ofHours(5))
+ assert(offsetOps.formatExternal(instant) === Some(s"2020-01-01
05:00:00.$frac"),
+ s"LTZ formatExternal for precision $p with +05:00")
+ }
+ }
+
private def checkOptionRoundtrip[T](
enc: AgnosticEncoder[Option[T]],
values: Seq[Option[T]]): Unit = {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
