This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new ae76eab12834 [SPARK-57486][SQL] Reuse AnyTimestampNanoType for
nanosecond-precision timestamp type checks
ae76eab12834 is described below
commit ae76eab128343014e69cd1f1b5e48f8becacac76
Author: Maxim Gekk <[email protected]>
AuthorDate: Tue Jun 16 22:10:32 2026 +0200
[SPARK-57486][SQL] Reuse AnyTimestampNanoType for nanosecond-precision
timestamp type checks
### What changes were proposed in this pull request?
[SPARK-57469](https://issues.apache.org/jira/browse/SPARK-57469) introduced
the `AnyTimestampNanoType` abstraction (an `AbstractDataType` plus the
`AnyTimestampNanoTypeExpression` extractor), but it was used in only one place.
Meanwhile, many sites across catalyst/core/hive still discriminated on the
nanosecond-precision timestamp types by spelling out the pair explicitly, e.g.:
```scala
case _: TimestampNTZNanosType | _: TimestampLTZNanosType => ...
t.isInstanceOf[TimestampNTZNanosType] ||
t.isInstanceOf[TimestampLTZNanosType]
```
This PR centralizes that check:
- Follow the existing `AnyTimeType` design: add a companion `abstract class
AnyTimestampNanoType extends DatetimeType { def precision: Int }` and make
`TimestampNTZNanosType` / `TimestampLTZNanosType` extend it (their `precision`
constructor `val` implements the abstract `def`). This lets the abstraction be
used as a plain type pattern `case _: AnyTimestampNanoType`.
- Simplify `AnyTimestampNanoType.acceptsType` and
`AnyTimestampNanoTypeExpression` to `isInstanceOf[AnyTimestampNanoType]`.
- Replace the explicit `TimestampNTZNanosType` + `TimestampLTZNanosType`
pair-checks across type-coercion, codegen, projections, hashing, and
data-source `supportDataType` checks with `case _: AnyTimestampNanoType`.
Sites that treat the two types differently (distinct converters/ops,
per-type instance construction, the `EXTRACT` error-message builder, parsers,
physical-type maps) are intentionally left unchanged.
This is a sub-task of
[SPARK-56822](https://issues.apache.org/jira/browse/SPARK-56822) (SPIP:
Timestamps with nanosecond precision).
### Why are the changes needed?
The pair-checks are duplicated logic that must be kept in sync whenever the
set of nanosecond timestamp types changes. Reusing the abstraction removes the
duplication and gives a single place that defines "is a nanosecond-precision
timestamp type".
### Does this PR introduce _any_ user-facing change?
No. The change is behavior-preserving: no user-facing change and no new
functionality. Because the nanosecond timestamp types are an unreleased preview
feature, changing their superclass has no binary-compatibility impact (verified
with MiMa).
### How was this patch tested?
- Compiled the affected modules: `build/sbt sql-api/compile
catalyst/compile sql/compile hive/compile avro/compile`.
- `./dev/scalastyle` and `dev/mima` pass.
- Existing nanosecond-timestamp test coverage continues to apply (no
behavior change).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
Closes #56540 from MaxGekk/nanos-reuse-anytimestamptype.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit bfe75dbddf090bda7a8cc84c7d91081b5de53986)
Signed-off-by: Max Gekk <[email protected]>
---
.../main/scala/org/apache/spark/sql/types/AbstractDataType.scala | 7 ++++++-
.../scala/org/apache/spark/sql/types/TimestampLTZNanosType.scala | 2 +-
.../scala/org/apache/spark/sql/types/TimestampNTZNanosType.scala | 2 +-
.../scala/org/apache/spark/sql/catalyst/expressions/Cast.scala | 6 ++----
.../sql/catalyst/expressions/InterpretedUnsafeProjection.scala | 4 ++--
.../org/apache/spark/sql/catalyst/expressions/ToStringBase.scala | 2 +-
.../spark/sql/catalyst/expressions/codegen/CodeGenerator.scala | 2 +-
.../catalyst/expressions/codegen/GenerateUnsafeProjection.scala | 6 +++---
.../spark/sql/catalyst/expressions/datetimeExpressions.scala | 4 ++--
.../scala/org/apache/spark/sql/catalyst/expressions/hash.scala | 2 +-
.../scala/org/apache/spark/sql/catalyst/expressions/literals.scala | 4 ++--
.../main/scala/org/apache/spark/sql/catalyst/util/TypeUtils.scala | 2 +-
.../scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala | 2 +-
.../main/scala/org/apache/spark/sql/types/DataTypeExpression.scala | 2 +-
.../apache/spark/sql/execution/vectorized/ColumnVectorUtils.java | 4 ++--
.../spark/sql/execution/vectorized/ConstantColumnVector.java | 2 +-
.../spark/sql/execution/vectorized/WritableColumnVector.java | 5 ++---
sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala | 2 +-
.../src/main/scala/org/apache/spark/sql/execution/Columnar.scala | 2 +-
.../apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala | 2 +-
.../spark/sql/execution/datasources/json/JsonFileFormat.scala | 2 +-
.../apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala | 2 +-
.../sql/execution/datasources/parquet/ParquetFileFormat.scala | 2 +-
.../apache/spark/sql/execution/datasources/v2/csv/CSVTable.scala | 4 ++--
.../apache/spark/sql/execution/datasources/v2/json/JsonTable.scala | 2 +-
.../apache/spark/sql/execution/datasources/v2/orc/OrcTable.scala | 2 +-
.../spark/sql/execution/datasources/v2/parquet/ParquetTable.scala | 2 +-
.../apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala | 2 +-
.../main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala | 2 +-
29 files changed, 43 insertions(+), 41 deletions(-)
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/types/AbstractDataType.scala
b/sql/api/src/main/scala/org/apache/spark/sql/types/AbstractDataType.scala
index 039d33a53bf4..9a27860ee125 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/AbstractDataType.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/AbstractDataType.scala
@@ -160,11 +160,16 @@ private[sql] object AnyTimestampType extends
AbstractDataType with Serializable
override private[sql] def simpleString = "(timestamp or timestamp without
time zone)"
}
+/**
+ * A nanosecond-precision timestamp type (`TIMESTAMP_LTZ(p)` /
`TIMESTAMP_NTZ(p)`, `p` in [7, 9]).
+ */
+private[sql] abstract class AnyTimestampNanoType extends DatetimeType
+
private[sql] object AnyTimestampNanoType extends AbstractDataType with
Serializable {
override private[sql] def defaultConcreteType: DataType =
TimestampNTZNanosType()
override private[sql] def acceptsType(other: DataType): Boolean =
- other.isInstanceOf[TimestampLTZNanosType] ||
other.isInstanceOf[TimestampNTZNanosType]
+ other.isInstanceOf[AnyTimestampNanoType]
override private[sql] def simpleString =
"(timestamp_ltz(p) or timestamp_ntz(p) with p in [7, 9])"
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampLTZNanosType.scala
b/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampLTZNanosType.scala
index 66bb238ba01f..beca6d33b707 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampLTZNanosType.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampLTZNanosType.scala
@@ -34,7 +34,7 @@ import org.apache.spark.sql.errors.DataTypeErrors
* @since 4.2.0
*/
@Unstable
-case class TimestampLTZNanosType(precision: Int) extends DatetimeType {
+case class TimestampLTZNanosType(precision: Int) extends AnyTimestampNanoType {
if (precision < TimestampLTZNanosType.MIN_PRECISION ||
precision > TimestampLTZNanosType.MAX_PRECISION) {
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampNTZNanosType.scala
b/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampNTZNanosType.scala
index 7ef496e6aaee..cf89e1451964 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampNTZNanosType.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampNTZNanosType.scala
@@ -34,7 +34,7 @@ import org.apache.spark.sql.errors.DataTypeErrors
* @since 4.2.0
*/
@Unstable
-case class TimestampNTZNanosType(precision: Int) extends DatetimeType {
+case class TimestampNTZNanosType(precision: Int) extends AnyTimestampNanoType {
if (precision < TimestampNTZNanosType.MIN_PRECISION ||
precision > TimestampNTZNanosType.MAX_PRECISION) {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
index fdb5ea110588..f3883047bed3 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
@@ -113,8 +113,7 @@ object Cast extends QueryErrorsBase {
case (DateType, TimestampNTZType) => true
case (TimestampType, TimestampNTZType) => true
- case (_: StringType, _: TimestampNTZNanosType) => true
- case (_: StringType, _: TimestampLTZNanosType) => true
+ case (_: StringType, _: AnyTimestampNanoType) => true
case (TimestampNTZType, _: TimestampNTZNanosType) => true
case (_: TimestampNTZNanosType, TimestampNTZType) => true
@@ -261,8 +260,7 @@ object Cast extends QueryErrorsBase {
case (DateType, TimestampNTZType) => true
case (TimestampType, TimestampNTZType) => true
- case (_: StringType, _: TimestampNTZNanosType) => true
- case (_: StringType, _: TimestampLTZNanosType) => true
+ case (_: StringType, _: AnyTimestampNanoType) => true
case (TimestampNTZType, _: TimestampNTZNanosType) => true
case (_: TimestampNTZNanosType, TimestampNTZType) => true
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedUnsafeProjection.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedUnsafeProjection.scala
index b01910d8b8d6..885e1b337c73 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedUnsafeProjection.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedUnsafeProjection.scala
@@ -260,7 +260,7 @@ object InterpretedUnsafeProjection {
case CalendarIntervalType =>
// We can't call setNullAt() for CalendarIntervalType, we call write
directly.
unsafeWriter
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType =>
+ case _: AnyTimestampNanoType =>
// We can't call setNullAt() for nanos timestamp types, we call write
directly.
unsafeWriter
case BooleanType | ByteType =>
@@ -308,7 +308,7 @@ object InterpretedUnsafeProjection {
@scala.annotation.tailrec
private def getElementSize(dataType: DataType): Int = dataType match {
case NullType | _: StringType | BinaryType | CalendarIntervalType |
VariantType |
- _: TimestampNTZNanosType | _: TimestampLTZNanosType |
+ _: AnyTimestampNanoType |
_: DecimalType | _: StructType | _: ArrayType | _: MapType => 8
case udt: UserDefinedType[_] =>
getElementSize(udt.sqlType)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
index 3a6ae2c2045f..65e0b125c225 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
@@ -238,7 +238,7 @@ trait ToStringBase { self: UnaryExpression with
TimeZoneAwareExpression =>
ctx.addReferenceObj("timestampNTZFormatter", timestampNTZFormatter),
timestampNTZFormatter.getClass)
(c, evPrim) => code"$evPrim = UTF8String.fromString($tf.format($c));"
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType =>
+ case _: AnyTimestampNanoType =>
// Route nanosecond timestamp cast-to-string through the Types
Framework: emit a runtime
// call into the ops reference object. The cast's session zone is
threaded into the lookup
// so LTZ carries it; NTZ is zone-independent (SPARK-57285).
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
index f70ad66a6563..2dfb590197b0 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
@@ -704,7 +704,7 @@ class CodegenContext extends Logging {
case CalendarIntervalType => s"$c1.compareTo($c2)"
// TimestampNanosVal exposes only `compareTo`; the AtomicType fallback
below emits
// `$c1.compare($c2)`, which would not resolve as a Java method call.
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType =>
s"$c1.compareTo($c2)"
+ case _: AnyTimestampNanoType => s"$c1.compareTo($c2)"
case NullType => "0"
case array: ArrayType =>
val elementType = array.elementType
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala
index cf2902175ea7..312b98093020 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala
@@ -38,7 +38,7 @@ object GenerateUnsafeProjection extends
CodeGenerator[Seq[Expression], UnsafePro
/** Returns true iff we support this data type. */
def canSupport(dataType: DataType): Boolean =
UserDefinedType.sqlType(dataType) match {
case NullType => true
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => true
+ case _: AnyTimestampNanoType => true
case _: AtomicType => true
case _: CalendarIntervalType => true
case t: StructType => t.forall(field => canSupport(field.dataType))
@@ -113,7 +113,7 @@ object GenerateUnsafeProjection extends
CodeGenerator[Seq[Expression], UnsafePro
// Can't call setNullAt() for DecimalType with precision larger
than 18.
s"$rowWriter.write($index, (Decimal) null, ${t.precision},
${t.scale});"
case CalendarIntervalType => s"$rowWriter.write($index,
(CalendarInterval) null);"
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType =>
+ case _: AnyTimestampNanoType =>
s"$rowWriter.write($index, (TimestampNanosVal) null);"
case _ => s"$rowWriter.setNullAt($index);"
}
@@ -191,7 +191,7 @@ object GenerateUnsafeProjection extends
CodeGenerator[Seq[Expression], UnsafePro
case t: DecimalType if t.precision > Decimal.MAX_LONG_DIGITS =>
s"$arrayWriter.write($index, (Decimal) null, ${t.precision},
${t.scale});"
case CalendarIntervalType => s"$arrayWriter.write($index,
(CalendarInterval) null);"
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType =>
+ case _: AnyTimestampNanoType =>
s"$arrayWriter.write($index, (TimestampNanosVal) null);"
case _ => s"$arrayWriter.setNull${elementOrOffsetSize}Bytes($index);"
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
index f7b019a87455..3db0e7a42509 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
@@ -491,7 +491,7 @@ case class SecondWithFractionNanos(child: Expression,
timeZoneId: Option[String]
override def dataType: DataType = DecimalType(11, 9)
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType =>
TypeCheckSuccess
+ case _: AnyTimestampNanoType => TypeCheckSuccess
case _ =>
DataTypeMismatch(
errorSubClass = "UNEXPECTED_INPUT_TYPE",
@@ -3464,7 +3464,7 @@ object DatePart {
// EXTRACT(SECOND) exposes the fractional digits, so
nanosecond-precision timestamps
// keep their sub-microsecond digits and widen the result to
DECIMAL(11, 9) instead of
// casting down to microseconds (SPARK-57340).
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType =>
+ case _: AnyTimestampNanoType =>
SecondWithFractionNanos(source)
case _ => SecondWithFraction(source)
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
index 795eaa22f0f1..3e6f6dbe22e0 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
@@ -546,7 +546,7 @@ abstract class HashExpression[E] extends Expression {
case ByteType | ShortType | IntegerType | DateType => genHashInt(input,
result)
case LongType | _: TimeType => genHashLong(input, result)
case TimestampType | TimestampNTZType => genHashTimestamp(input, result)
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType =>
+ case _: AnyTimestampNanoType =>
genHashTimestampNanos(input, result)
case FloatType => genHashFloat(input, result)
case DoubleType => genHashDouble(input, result)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
index e444929b216a..75526db24ddf 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
@@ -196,7 +196,7 @@ object Literal {
!v.isInstanceOf[MapData] &&
!v.isInstanceOf[InternalRow] &&
dataType.existsRecursively { t =>
- t.isInstanceOf[TimestampNTZNanosType] ||
t.isInstanceOf[TimestampLTZNanosType]
+ t.isInstanceOf[AnyTimestampNanoType]
}
}
@@ -524,7 +524,7 @@ case class Literal (value: Any, dataType: DataType) extends
LeafExpression {
case (null, _) => JNull
case (i: Int, DateType) => JString(toString)
case (l: Long, TimestampType | _: TimeType) => JString(toString)
- case (_: TimestampNanosVal, _: TimestampNTZNanosType | _:
TimestampLTZNanosType) =>
+ case (_: TimestampNanosVal, _: AnyTimestampNanoType) =>
JString(toString)
case (other, _) => JString(other.toString)
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TypeUtils.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TypeUtils.scala
index f82b6c58ddf4..c6792af9f0ea 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TypeUtils.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TypeUtils.scala
@@ -141,7 +141,7 @@ object TypeUtils extends QueryErrorsBase {
private def containsTimestampNanosType(dataType: DataType): Boolean = {
dataType.existsRecursively {
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => true
+ case _: AnyTimestampNanoType => true
case _ => false
}
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala
index 470b0e1f6a6b..5a835e5e7ee4 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala
@@ -200,7 +200,7 @@ object UnsafeRowUtils {
def avoidSetNullAt(dt: DataType): Boolean = dt match {
case t: DecimalType if t.precision > Decimal.MAX_LONG_DIGITS => true
case CalendarIntervalType => true
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => true
+ case _: AnyTimestampNanoType => true
case _ => false
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataTypeExpression.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataTypeExpression.scala
index a7f50c0663fa..6cb6b7d429d8 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataTypeExpression.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataTypeExpression.scala
@@ -84,7 +84,7 @@ private[sql] object AnyTimestampTypeExpression {
private[sql] object AnyTimestampNanoTypeExpression {
def unapply(e: Expression): Boolean =
- e.dataType.isInstanceOf[TimestampLTZNanosType] ||
e.dataType.isInstanceOf[TimestampNTZNanosType]
+ e.dataType.isInstanceOf[AnyTimestampNanoType]
}
private[sql] object DecimalExpression {
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java
index d07a345f7dfb..2e57d37c9f80 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java
@@ -176,7 +176,7 @@ public class ColumnVectorUtils {
private static void appendValue(WritableColumnVector dst, DataType t, Object
o) {
if (o == null) {
if (t instanceof CalendarIntervalType || t instanceof VariantType ||
- t instanceof TimestampNTZNanosType || t instanceof
TimestampLTZNanosType) {
+ t instanceof AnyTimestampNanoType) {
dst.appendStruct(true);
} else {
dst.appendNull();
@@ -224,7 +224,7 @@ public class ColumnVectorUtils {
dst.appendStruct(false);
dst.getChild(0).appendByteArray(v.getValue(), 0, v.getValue().length);
dst.getChild(1).appendByteArray(v.getMetadata(), 0,
v.getMetadata().length);
- } else if (t instanceof TimestampNTZNanosType || t instanceof
TimestampLTZNanosType) {
+ } else if (t instanceof AnyTimestampNanoType) {
TimestampNanosVal v = (TimestampNanosVal) o;
dst.appendStruct(false);
dst.getChild(0).appendLong(v.epochMicros);
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java
index fd99df0fb9f0..a579abf2565e 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java
@@ -74,7 +74,7 @@ public class ConstantColumnVector extends ColumnVector {
this.childData[0] = new ConstantColumnVector(1, DataTypes.IntegerType);
this.childData[1] = new ConstantColumnVector(1, DataTypes.IntegerType);
this.childData[2] = new ConstantColumnVector(1, DataTypes.LongType);
- } else if (type instanceof TimestampNTZNanosType || type instanceof
TimestampLTZNanosType) {
+ } else if (type instanceof AnyTimestampNanoType) {
// Two columns. EpochMicros as Long. NanosWithinMicro as Short.
this.childData = new ConstantColumnVector[2];
this.childData[0] = new ConstantColumnVector(1, DataTypes.LongType);
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
index 016c70b712dc..fab67965d701 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
@@ -783,8 +783,7 @@ public abstract class WritableColumnVector extends
ColumnVector {
for (WritableColumnVector c: childColumns) {
if (c.type instanceof StructType || c.type instanceof VariantType
|| c.type instanceof CalendarIntervalType
- || c.type instanceof TimestampNTZNanosType
- || c.type instanceof TimestampLTZNanosType) {
+ || c.type instanceof AnyTimestampNanoType) {
c.appendStruct(true);
} else {
c.appendNull();
@@ -1089,7 +1088,7 @@ public abstract class WritableColumnVector extends
ColumnVector {
this.childColumns[0] = reserveNewColumn(capacity, DataTypes.IntegerType);
this.childColumns[1] = reserveNewColumn(capacity, DataTypes.IntegerType);
this.childColumns[2] = reserveNewColumn(capacity, DataTypes.LongType);
- } else if (type instanceof TimestampNTZNanosType || type instanceof
TimestampLTZNanosType) {
+ } else if (type instanceof AnyTimestampNanoType) {
// Two columns. EpochMicros as Long. NanosWithinMicro as Short.
this.childColumns = new WritableColumnVector[2];
this.childColumns[0] = reserveNewColumn(capacity, DataTypes.LongType);
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
index 7019e925080c..6f410337b796 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
@@ -110,7 +110,7 @@ private[sql] object AvroUtils extends Logging {
case _: GeometryType | _: GeographyType => false
// Nanosecond-capable timestamps are not yet supported by this datasource.
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => false
+ case _: AnyTimestampNanoType => false
case _: AtomicType => true
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
index c9d766308b23..81fc5d2c1f73 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
@@ -285,7 +285,7 @@ private object RowToColumnConverter {
if (nullable) {
dataType match {
case CalendarIntervalType | VariantType => new
StructNullableTypeConverter(core)
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType =>
+ case _: AnyTimestampNanoType =>
new StructNullableTypeConverter(core)
case st: StructType => new StructNullableTypeConverter(core)
case _ => new BasicNullableTypeConverter(core)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
index 26ad735e89aa..f4515c7530a6 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
@@ -168,7 +168,7 @@ case class CSVFileFormat() extends TextBasedFileFormat with
DataSourceRegister {
case _: GeometryType | _: GeographyType => false
// Nanosecond-capable timestamps are not yet supported by this datasource.
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => false
+ case _: AnyTimestampNanoType => false
case _: AtomicType => true
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
index 2ddce6286729..2d3424847ff8 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
@@ -128,7 +128,7 @@ case class JsonFileFormat() extends TextBasedFileFormat
with DataSourceRegister
case _: GeometryType | _: GeographyType => false
// Nanosecond-capable timestamps are not yet supported by this datasource.
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => false
+ case _: AnyTimestampNanoType => false
case _: AtomicType => true
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
index 8b926f51efc3..7065693e59ca 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
@@ -252,7 +252,7 @@ class OrcFileFormat
case _: GeometryType | _: GeographyType => false
// Nanosecond-capable timestamps are not yet supported by this datasource.
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => false
+ case _: AnyTimestampNanoType => false
case _: AtomicType => true
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
index 669d826ddac4..2b32c4e0d056 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
@@ -417,7 +417,7 @@ class ParquetFileFormat
case g: GeographyType => GeographyType.isSridSupported(g.srid)
// Nanosecond-capable timestamps are not yet supported by this datasource.
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => false
+ case _: AnyTimestampNanoType => false
case _: AtomicType | _: NullType => true
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVTable.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVTable.scala
index a1d9f2f7c90b..8fdaad44376f 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVTable.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVTable.scala
@@ -26,7 +26,7 @@ import
org.apache.spark.sql.connector.write.{LogicalWriteInfo, Write, WriteBuild
import org.apache.spark.sql.execution.datasources.FileFormat
import org.apache.spark.sql.execution.datasources.csv.CSVDataSource
import org.apache.spark.sql.execution.datasources.v2.FileTable
-import org.apache.spark.sql.types.{AtomicType, DataType, GeographyType,
GeometryType, StructType, TimestampLTZNanosType, TimestampNTZNanosType,
UserDefinedType}
+import org.apache.spark.sql.types.{AnyTimestampNanoType, AtomicType, DataType,
GeographyType, GeometryType, StructType, UserDefinedType}
import org.apache.spark.sql.util.CaseInsensitiveStringMap
case class CSVTable(
@@ -64,7 +64,7 @@ case class CSVTable(
case _: GeometryType | _: GeographyType => false
// Nanosecond-capable timestamps are not yet supported by this datasource.
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => false
+ case _: AnyTimestampNanoType => false
case _: AtomicType => true
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonTable.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonTable.scala
index 2edc5702e0b8..0612041c1a72 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonTable.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonTable.scala
@@ -62,7 +62,7 @@ case class JsonTable(
case _: GeometryType | _: GeographyType => false
// Nanosecond-capable timestamps are not yet supported by this datasource.
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => false
+ case _: AnyTimestampNanoType => false
case _: AtomicType => true
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcTable.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcTable.scala
index e595a680d2e6..24cd5e60321e 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcTable.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcTable.scala
@@ -54,7 +54,7 @@ case class OrcTable(
case _: GeometryType | _: GeographyType => false
// Nanosecond-capable timestamps are not yet supported by this datasource.
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => false
+ case _: AnyTimestampNanoType => false
case _: AtomicType => true
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetTable.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetTable.scala
index 9c3b05504363..19974f613cc0 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetTable.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetTable.scala
@@ -56,7 +56,7 @@ case class ParquetTable(
case g: GeographyType => GeographyType.isSridSupported(g.srid)
// Nanosecond-capable timestamps are not yet supported by this datasource.
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => false
+ case _: AnyTimestampNanoType => false
case _: AtomicType => true
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala
index f32b915fc054..1ce253e76f45 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala
@@ -135,7 +135,7 @@ case class XmlFileFormat() extends TextBasedFileFormat with
DataSourceRegister {
case _: GeometryType | _: GeographyType => false
// Nanosecond-capable timestamps are not yet supported by this datasource.
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => false
+ case _: AnyTimestampNanoType => false
case _: AtomicType => true
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
index b8961e0ee84e..6c3fbb5a8314 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
@@ -198,7 +198,7 @@ case class OrcFileFormat() extends FileFormat
case _: AnsiIntervalType => false
case _: TimeType => false
// Nanosecond-capable timestamps are not yet supported by this datasource.
- case _: TimestampNTZNanosType | _: TimestampLTZNanosType => false
+ case _: AnyTimestampNanoType => false
case _: AtomicType => true
case st: StructType => st.forall { f => supportDataType(f.dataType) }
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
