This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 03c44cc47684 [SPARK-57386][SQL] Render nanosecond timestamp types in
HiveResult through the Types Framework
03c44cc47684 is described below
commit 03c44cc4768404a565606cd368bbd2c92561e81e
Author: Maxim Gekk <[email protected]>
AuthorDate: Tue Jun 16 10:27:05 2026 +0200
[SPARK-57386][SQL] Render nanosecond timestamp types in HiveResult through
the Types Framework
### What changes were proposed in this pull request?
Unify nanosecond timestamp (`TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)`)
rendering in `HiveResult` onto the Types Framework, removing the inline
duplicate renderer.
- `TimestampNanosTypeApiOps`: remove the `formatExternal(value, nested) =
None` override so the Hive path shares each subclass's single-arg
`formatExternal` renderer (the same one Row JSON uses). `nested` does not
affect timestamp formatting.
- `HiveResult.toHiveStringDefault`: remove the inline
`TimestampLTZNanosType` / `TimestampNTZNanosType` cases. The legacy path keeps
no nanos handling, so a nanos value that somehow reaches it (only possible with
the framework off, which the gating forbids) is unsupported rather than
silently rendered.
- `TypeApiOps`: update the two-arg `formatExternal` scaladoc to reflect
that Hive now shares the single-arg renderer.
### Why are the changes needed?
The nanosecond timestamp types are a Types Framework feature, implemented
solely through it. External-value rendering for the framework is centralized in
`TypeApiOps.formatExternal`, which already backs Row JSON (`Row.json` /
`Row.prettyJson`).
The nanos ops previously overrode the two-arg `formatExternal(value,
nested)` to return `None`, so `HiveResult` rendered nanos through inline
pattern-matching in `toHiveStringDefault`. That duplicated the formatter logic
and was documented in code as a temporary split "until nanos external rendering
is unified across the zone-less (Row JSON) and zone-aware (Hive) paths".
The types are gated by `timestampNanosTypesEnabled =
timestampNanosTypes.enabled && types.framework.enabled`, so a nanos column
cannot exist while the framework is off; the inline cases are therefore dead
code in that mode and redundant when the framework is on.
### Does this PR introduce _any_ user-facing change?
No. Nanos Hive output is identical: zone-aware LTZ, zone-independent NTZ,
precision flooring, and trailing-zero trimming are all unchanged. Both the old
inline path and the framework path ultimately call the same
`TimestampFormatter` methods; the pre-flooring in the inline path is a no-op
because values reaching Hive come from `executeCollectPublic()`, whose internal
value is already stored floored to the column precision.
### How was this patch tested?
Existing `HiveResultSuite` SPARK-57257 tests cover precision 7/8/9,
pre-1970 epochs, nested arrays/maps/structs, NULLs, and session-zone vs
zone-independent rendering, and now exercise the framework path. Also ran
`TimestampNanosTypeOpsSuite` and `RowJsonSuite`. All pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56513 from MaxGekk/nanos-update-HiveResult.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit f49ca1497fe8d1a50e58f2f11457640369aa4568)
Signed-off-by: Max Gekk <[email protected]>
---
.../spark/sql/types/ops/TimestampNanosTypeApiOps.scala | 18 +++++++-----------
.../org/apache/spark/sql/types/ops/TypeApiOps.scala | 8 ++++----
.../org/apache/spark/sql/execution/HiveResult.scala | 10 ----------
3 files changed, 11 insertions(+), 25 deletions(-)
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
b/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
index d9b3a431f1ec..32aac4dcfd9c 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
@@ -42,6 +42,13 @@ import org.apache.spark.unsafe.types.TimestampNanosVal
* resolved session zone, while zone-less callers (Row JSON via
formatExternal) accept the
* default, the session-local time zone config.
*
+ * External-value rendering (SPARK-57386): both consumers go through the
single-arg
+ * [[formatExternal]] each subclass overrides - Row JSON (Row.json /
Row.prettyJson) calls it
+ * directly, and the Hive result path (HiveResult.toHiveString) calls the
two-arg overload whose
+ * default delegates to it (`nested` does not affect timestamp formatting). So
both render the
+ * external value at the column precision rather than truncating to
microseconds via the legacy
+ * path.
+ *
* Dataset encoders are wired here to the precision-aware leaves added by
SPARK-57033
* (LocalDateTimeNanosEncoder / InstantNanosEncoder), so that turning on the
Types Framework
* matches the legacy RowEncoder.encoderForDataTypeDefault behavior rather
than regressing it.
@@ -58,17 +65,6 @@ abstract class TimestampNanosTypeApiOps extends TypeApiOps
with DataTypeErrorsBa
// ==================== String Formatting ====================
- // Row JSON (Row.json / Row.prettyJson) holds the external Row value
(java.time.Instant for LTZ,
- // java.time.LocalDateTime for NTZ); each subclass overrides the single-arg
formatExternal to
- // render it through the same formatter as its zone-aware cast-to-string, so
Row JSON shows the
- // nanosecond value rather than silently truncating to microseconds via the
legacy path.
-
- // The Hive result path (HiveResult.toHiveString) renders nanosecond
timestamps through its own
- // zone-aware default formatter, so return None here to fall through to it
rather than to the
- // subclass single-arg rendering. This is a temporary split until nanos
external rendering is
- // unified across the zone-less (Row JSON) and zone-aware (Hive) paths.
- override def formatExternal(value: Any, nested: Boolean): Option[String] =
None
-
override def toSQLValue(v: Any): String = s"$sqlTypeName '${format(v)}'"
// ==================== Row Encoding ====================
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
b/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
index ab46e34c91e3..c76551aacd84 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
@@ -145,10 +145,10 @@ trait TypeApiOps extends Serializable {
* Renders an external value for Hive-style output
(HiveResult.toHiveString). `nested` indicates
* whether the value appears inside an array/map/struct, which may
format/quote it differently.
* Semantics mirror the single-arg overload: Some(s) is used directly, None
falls back to
- * HiveResult's zone-aware legacy rendering. The default delegates to the
single-arg overload;
- * override it separately when the two consumers need different behavior
(e.g. the nanosecond
- * timestamp types render the external value on the zone-less Row JSON path
but return None here
- * so the zone-aware Hive path renders them through its own formatter).
+ * HiveResult's zone-aware legacy rendering. The default delegates to the
single-arg overload,
+ * so Hive shares the same renderer as Row JSON (e.g. the nanosecond
timestamp types render the
+ * external Instant/LocalDateTime at the column precision on both paths).
Override it separately
+ * only when the two consumers need different behavior.
*/
def formatExternal(value: Any, nested: Boolean): Option[String] =
formatExternal(value)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
index 25170da800ae..671451087435 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
@@ -131,16 +131,6 @@ object HiveResult extends SQLConfHelper {
case (t: Timestamp, TimestampType) => formatters.timestamp.format(t)
case (i: Instant, TimestampType) => formatters.timestamp.format(i)
case (l: LocalDateTime, TimestampNTZType) => formatters.timestamp.format(l)
- // Nanosecond-precision timestamps. The external values are `Instant`
(LTZ) and
- // `LocalDateTime` (NTZ); convert to the physical `TimestampNanosVal` at
the column precision
- // and render via the same formatter methods as the cast-to-string path
(SPARK-57256), so the
- // output stays consistent. LTZ uses the session zone; NTZ is
zone-independent.
- case (i: Instant, t: TimestampLTZNanosType) =>
- formatters.timestamp.formatNanos(
- DateTimeUtils.instantToTimestampNanos(i, t.precision), t.precision)
- case (l: LocalDateTime, t: TimestampNTZNanosType) =>
- formatters.timestamp.formatWithoutTimeZoneNanos(
- DateTimeUtils.localDateTimeToTimestampNanos(l, t.precision),
t.precision)
case (bin: Array[Byte], BinaryType) => binaryFormatter(bin)
case (decimal: java.math.BigDecimal, DecimalType()) =>
decimal.toPlainString
case (n, _: NumericType) => n.toString
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
