This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 6898d7f3824e [SPARK-57259][SQL][TEST] Add nanosecond timestamp types
to DataTypeTestUtils type sets
6898d7f3824e is described below
commit 6898d7f3824e8f7abec66e3fddcc0aff4aa183c2
Author: Maxim Gekk <[email protected]>
AuthorDate: Tue Jun 9 11:29:39 2026 +0200
[SPARK-57259][SQL][TEST] Add nanosecond timestamp types to
DataTypeTestUtils type sets
### What changes were proposed in this pull request?
Add `TimestampLTZNanosType` and `TimestampNTZNanosType` (at min and max
precision) to the shared type sets in `DataTypeTestUtils` (`ordered`,
`atomicTypes`, and the derived `propertyCheckSupported` and
`atomicArrayTypes`), mirroring the existing `timeTypes` pattern. Also refresh a
stale comment in `OrderingSuite` now that the generic `atomicTypes` loop covers
the nanosecond timestamp types.
### Why are the changes needed?
`TimestampLTZNanosType` and `TimestampNTZNanosType` extend `DatetimeType`
-> `AtomicType` but were absent from the `DataTypeTestUtils` type sets. Adding
them triggers broader generic test coverage across `OrderingSuite`,
`PredicateSuite`, `ConditionalExpressionSuite`, `ArithmeticExpressionSuite`
(LEAST/GREATEST), `SortSuite`, `RandomDataGeneratorSuite`, and the Cast suites,
exposing any gaps in the nanosecond timestamp type infrastructure.
**This PR depends on SPARK-57317 (#56371)**, which fixes `Literal.create`
for external nanosecond timestamp values; without it the new coverage in
`PredicateSuite` ("IN with different types") fails. CI here is expected to be
red until #56371 is merged and this branch is rebased.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Ran the affected suites with this change on top of SPARK-57317:
`OrderingSuite`, `RandomDataGeneratorSuite`, `PredicateSuite`,
`ConditionalExpressionSuite`, `ArithmeticExpressionSuite`, `CastSuiteBase`,
`CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `SortSuite`, `UnsafeRowSuite`.
All passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56372 from MaxGekk/spark-57259-test-sets.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit ac4457e4e5c54efacdcd3463f9284536a19f7427)
Signed-off-by: Max Gekk <[email protected]>
---
.../spark/sql/catalyst/expressions/OrderingSuite.scala | 7 ++++---
.../scala/org/apache/spark/sql/types/DataTypeTestUtils.scala | 12 ++++++++++--
2 files changed, 14 insertions(+), 5 deletions(-)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala
index 807849a0f1ea..091aa0e0997c 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala
@@ -183,9 +183,10 @@ class OrderingSuite extends SparkFunSuite with
ExpressionEvalHelper {
GenerateOrdering.generate(Array.fill(5000)(sortOrder).toImmutableArraySeq)
}
- // SPARK-57103: ordering for nanosecond timestamp types. Not driven by the
generic
- // `atomicTypes` loop above because `RandomDataGenerator` does not yet
support the new
- // types (tracked separately in SPARK-57034); we hand-roll edge cases here
instead.
+ // SPARK-57103: ordering for nanosecond timestamp types. The generic
`atomicTypes` loop above
+ // now covers these types (SPARK-57034 added `RandomDataGenerator` support
and SPARK-57259 added
+ // them to `DataTypeTestUtils`), but we still hand-roll the edge cases below
(Long boundaries,
+ // tie-breakers, null ordering) that random generation is unlikely to hit.
private def compareNanos(
dataType: AtomicType,
a: TimestampNanosVal,
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeTestUtils.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeTestUtils.scala
index fc011647fc8d..a33b6010f4b6 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeTestUtils.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeTestUtils.scala
@@ -73,6 +73,12 @@ object DataTypeTestUtils {
TimeType(TimeType.MIN_PRECISION),
TimeType(TimeType.MAX_PRECISION))
+ val timestampNanosTypes: Seq[DatetimeType] = Seq(
+ TimestampLTZNanosType(TimestampLTZNanosType.MIN_PRECISION),
+ TimestampLTZNanosType(TimestampLTZNanosType.MAX_PRECISION),
+ TimestampNTZNanosType(TimestampNTZNanosType.MIN_PRECISION),
+ TimestampNTZNanosType(TimestampNTZNanosType.MAX_PRECISION))
+
val unsafeRowMutableFieldTypes: Seq[DataType] = Seq(
NullType,
BooleanType,
@@ -101,7 +107,8 @@ object DataTypeTestUtils {
TimestampNTZType,
DateType,
StringType,
- BinaryType) ++ dayTimeIntervalTypes ++ yearMonthIntervalTypes ++ timeTypes
+ BinaryType) ++ dayTimeIntervalTypes ++ yearMonthIntervalTypes ++ timeTypes
++
+ timestampNanosTypes
/**
* All the types that we can use in a property check
@@ -117,7 +124,8 @@ object DataTypeTestUtils {
DateType,
StringType,
TimestampType,
- TimestampNTZType) ++ dayTimeIntervalTypes ++ yearMonthIntervalTypes ++
timeTypes
+ TimestampNTZType) ++ dayTimeIntervalTypes ++ yearMonthIntervalTypes ++
timeTypes ++
+ timestampNanosTypes
/**
* Instances of [[ArrayType]] for all [[AtomicType]]s. Arrays of these types
may contain null.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
