This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 9a230895724a [SPARK-57164][SQL][TEST] Add parser test coverage for
nanosecond-capable timestamp types across data-type string entry points
9a230895724a is described below
commit 9a230895724a9bf16b122505147ec6e5fc160f18
Author: Maxim Gekk <[email protected]>
AuthorDate: Tue Jun 16 01:14:32 2026 +0200
[SPARK-57164][SQL][TEST] Add parser test coverage for nanosecond-capable
timestamp types across data-type string entry points
### What changes were proposed in this pull request?
Test-only changes (sub-task of
[SPARK-56822](https://issues.apache.org/jira/browse/SPARK-56822), SPIP:
Timestamps with nanosecond precision) adding focused coverage that the
nanosecond-capable timestamp spellings (`TIMESTAMP_NTZ(p)`, `TIMESTAMP_LTZ(p)`,
and the `TIMESTAMP(p) WITH[OUT] [LOCAL] TIME ZONE` aliases, `p` in `[7, 9]`)
parse consistently across the public string-to-DataType entry points. All new
assertions are gated by `spark.sql.timestampNanosTypes.enabled`.
Coverage added:
- Catalyst `DataTypeParserSuite`: `DataType.fromDDL`, `StructType.fromDDL`,
`StructType.add(name, String)`, and `DataType.parseTypeWithFallback` (DDL path
+ JSON fallback, the guard against the ANTLR and JSON parser families drifting
apart).
- Catalyst `ExpressionParserSuite`: `CAST` / `TRY_CAST` to nanos types.
- Catalyst `DDLParserSuite`: `CREATE TABLE`, `ALTER TABLE ADD COLUMNS`,
`ALTER COLUMN ... TYPE`, and a column `DEFAULT` declared with a nanos type.
- Catalyst `DataTypeSuite`: `typeName` re-parse leg of the Family B (JSON)
round-trip.
- `ColumnExpressionSuite`: `Column.cast(String)` / `try_cast(String)` and
`sessionState.sqlParser.parseDataType(String)`.
- `DataFrameReaderWriterSuite` / `DataStreamReaderWriterSuite`:
`schema(String)`.
- `JsonFunctionsSuite` / `CsvFunctionsSuite` / `XmlFunctionsSuite`:
`from_json` / `from_csv` / `from_xml` with nanos DDL schema strings, asserting
the string resolves to the nanos type but the datasource rejects it at
execution (JSON/CSV: `UNSUPPORTED_DATATYPE`; XML: `MALFORMED_RECORD_IN_PARSING`
in FAILFAST mode).
### Why are the changes needed?
Spark parses data-type strings through two independent parser families
(ANTLR `DataTypeAstBuilder` and the hand-maintained JSON `nameToType` in
`DataType.scala`), reached through many distinct public surfaces. The nanos
parsing was previously exercised mainly via `CatalystSqlParser.parseDataType`;
the other public entry points had no explicit assertions, so a regression on
any one of them (or drift between the two families) could go unnoticed.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
Ran the affected suites locally; all pass:
- `build/sbt 'catalyst/testOnly *DataTypeParserSuite *DataTypeSuite
*ExpressionParserSuite *DDLParserSuite'`
- `build/sbt 'sql/testOnly *ColumnExpressionSuite
*DataFrameReaderWriterSuite *DataStreamReaderWriterSuite *JsonFunctionsSuite
*CsvFunctionsSuite *XmlFunctionsSuite'`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56514 from MaxGekk/nanos-parser-tests.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit 3256fca131fa40a516ce68309c0ec2fed534d974)
Signed-off-by: Max Gekk <[email protected]>
---
.../spark/sql/catalyst/parser/DDLParserSuite.scala | 47 +++++++-
.../sql/catalyst/parser/DataTypeParserSuite.scala | 121 +++++++++++++++++++++
.../catalyst/parser/ExpressionParserSuite.scala | 32 +++++-
.../org/apache/spark/sql/types/DataTypeSuite.scala | 10 ++
.../apache/spark/sql/ColumnExpressionSuite.scala | 35 ++++++
.../org/apache/spark/sql/CsvFunctionsSuite.scala | 26 +++++
.../org/apache/spark/sql/JsonFunctionsSuite.scala | 29 ++++-
.../org/apache/spark/sql/XmlFunctionsSuite.scala | 29 +++++
.../test/DataStreamReaderWriterSuite.scala | 37 +++++++
.../sql/test/DataFrameReaderWriterSuite.scala | 36 +++++-
10 files changed, 397 insertions(+), 5 deletions(-)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
index 7907d80e8b0c..2e8132926dcd 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
@@ -19,17 +19,18 @@ package org.apache.spark.sql.catalyst.parser
import java.util.Locale
-import org.apache.spark.SparkThrowable
+import org.apache.spark.{SparkException, SparkThrowable}
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.analysis._
import org.apache.spark.sql.catalyst.expressions.{Add, And, EqualTo,
GreaterThan, Hex, Literal}
import org.apache.spark.sql.catalyst.plans.logical._
+import
org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.connector.catalog.IdentityColumnSpec
import
org.apache.spark.sql.connector.catalog.TableChange.ColumnPosition.{after, first}
import org.apache.spark.sql.connector.expressions.{ApplyTransform,
BucketTransform, ClusterByTransform, DaysTransform, FieldReference,
HoursTransform, IdentityTransform, LiteralValue, MonthsTransform, Transform,
YearsTransform}
import org.apache.spark.sql.connector.expressions.LogicalExpressions.bucket
import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.{DataType, Decimal, IntegerType, LongType,
StringType, StructType, TimestampType}
+import org.apache.spark.sql.types.{DataType, Decimal, IntegerType, LongType,
StringType, StructType, TimestampLTZNanosType, TimestampNTZNanosType,
TimestampType}
import org.apache.spark.storage.StorageLevelMapper
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
@@ -90,6 +91,48 @@ class DDLParserSuite extends AnalysisTest {
parameters = Map("error" -> "':'", "hint" -> ""))
}
+ test("SPARK-57164: nanos timestamp types in CREATE TABLE / ALTER TABLE
columns") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" ->
TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ // CREATE TABLE column.
+ val created = parsePlan(s"CREATE TABLE t (c $spelling) USING
parquet")
+ assert(created.asInstanceOf[CreateTable].columns.head.dataType ===
expected)
+ // ALTER TABLE ... ADD COLUMNS.
+ val added = parsePlan(s"ALTER TABLE t ADD COLUMNS (c $spelling)")
+ assert(added.asInstanceOf[AddColumns].columnsToAdd.head.dataType
=== expected)
+ // ALTER TABLE ... ALTER COLUMN ... TYPE.
+ val altered = parsePlan(s"ALTER TABLE t ALTER COLUMN c TYPE
$spelling")
+ assert(altered.asInstanceOf[AlterColumns].specs.head.newDataType
=== Some(expected))
+ }
+ }
+ // A column DEFAULT declared with a nanos type and a nanos typed-literal
default.
+ val withDefault = parsePlan(
+ "CREATE TABLE t (c TIMESTAMP_NTZ(9) " +
+ "DEFAULT TIMESTAMP_NTZ '2020-01-01 00:00:00.123456789') USING
parquet")
+ val colDef = withDefault.asInstanceOf[CreateTable].columns.head
+ assert(colDef.dataType === TimestampNTZNanosType(9))
+ assert(colDef.defaultValue.isDefined)
+ }
+ // With the preview flag off, a nanos column type is rejected.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ checkError(
+ exception = intercept[SparkException] {
+ parsePlan("CREATE TABLE t (c TIMESTAMP_NTZ(9)) USING parquet")
+ },
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+
test("create/replace table - with IF NOT EXISTS") {
val sql = "CREATE TABLE IF NOT EXISTS my_tab(a INT, b STRING) USING
parquet"
testCreateOrReplaceDdl(
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
index 4bb243bfe464..084bdbc460b7 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.catalyst.parser
import org.apache.spark.{SparkException, SparkFunSuite}
import org.apache.spark.sql.catalyst.plans.SQLHelper
+import
org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.TimestampTypes
import org.apache.spark.sql.types._
@@ -377,4 +378,124 @@ class DataTypeParserSuite extends SparkFunSuite with
SQLHelper {
parameters = Map("error" -> "'-'", "hint" -> ""))
}
}
+
+ test("SPARK-57164: nanos timestamp types via DataType.fromDDL and
StructType.fromDDL") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" ->
TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ val expectedStruct = new StructType().add("c", expected)
+ assert(DataType.fromDDL(s"c $spelling") === expectedStruct)
+ assert(StructType.fromDDL(s"c $spelling") === expectedStruct)
+ // A bare type string (no column name) resolves directly to the
nanos type.
+ assert(DataType.fromDDL(spelling) === expected)
+ }
+ // Bare TIMESTAMP(p) follows the session default timestamp type.
+ withSQLConf(SQLConf.TIMESTAMP_TYPE.key ->
TimestampTypes.TIMESTAMP_NTZ.toString) {
+ assert(DataType.fromDDL(s"c TIMESTAMP($p)") ===
+ new StructType().add("c", TimestampNTZNanosType(p)))
+ }
+ withSQLConf(SQLConf.TIMESTAMP_TYPE.key ->
TimestampTypes.TIMESTAMP_LTZ.toString) {
+ assert(DataType.fromDDL(s"c TIMESTAMP($p)") ===
+ new StructType().add("c", TimestampLTZNanosType(p)))
+ }
+ }
+ }
+ }
+
+ test("SPARK-57164: fromDDL maps TIMESTAMP_*(6) to GA micro types and rejects
bad precision") {
+ // Precision 6 maps to the GA micro types regardless of the preview flag.
+ Seq("true", "false").foreach { flag =>
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> flag) {
+ assert(DataType.fromDDL("c TIMESTAMP_NTZ(6)") ===
+ new StructType().add("c", TimestampNTZType))
+ assert(DataType.fromDDL("c TIMESTAMP_LTZ(6)") ===
+ new StructType().add("c", TimestampType))
+ }
+ }
+ // Out-of-range precision surfaces as INVALID_TIMESTAMP_PRECISION. The
bare-type spelling is
+ // used so the primary parser raises the precision error (a
SparkThrowable), which
+ // parseTypeWithFallback rethrows in preference to the fallback's parse
error.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ Seq("TIMESTAMP_NTZ" -> "TIMESTAMP_NTZ", "TIMESTAMP_LTZ" ->
"TIMESTAMP_LTZ").foreach {
+ case (spelling, errorType) =>
+ Seq(0, 5, 10).foreach { p =>
+ checkError(
+ exception = intercept[SparkException] {
+ DataType.fromDDL(s"$spelling($p)")
+ },
+ condition = "INVALID_TIMESTAMP_PRECISION",
+ parameters = Map("precision" -> p.toString, "type" -> errorType))
+ }
+ }
+ }
+ // With the flag off, nanos precisions are rejected with
FEATURE_NOT_ENABLED.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ Seq("TIMESTAMP_NTZ(9)", "TIMESTAMP_LTZ(7)").foreach { spelling =>
+ checkError(
+ exception = intercept[SparkException] {
+ DataType.fromDDL(spelling)
+ },
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+ }
+
+ test("SPARK-57164: nanos timestamp types via StructType.add(name,
dataTypeString)") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" ->
TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ assert(new StructType().add("c", spelling) ===
+ new StructType().add("c", expected))
+ }
+ }
+ }
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ checkError(
+ exception = intercept[SparkException] {
+ new StructType().add("c", "TIMESTAMP_NTZ(9)")
+ },
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+
+ test("SPARK-57164: parseTypeWithFallback resolves nanos types via DDL and
JSON fallback") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ // DDL path: the primary parser (fromDDL) succeeds and the fallback is
never consulted.
+ assert(
+ DataType.parseTypeWithFallback(
+ s"c TIMESTAMP_NTZ($p)",
+ DataType.fromDDL,
+ fallbackParser = DataType.fromJson) ===
+ new StructType().add("c", TimestampNTZNanosType(p)))
+ // JSON fallback path: the DDL parser fails on the quoted JSON name,
so the JSON parser
+ // handles it. This is the single best guard against Family A (DDL)
and Family B (JSON)
+ // drifting apart.
+ val jsonName = TimestampLTZNanosType(p).typeName // "timestamp_ltz(p)"
+ assert(
+ DataType.parseTypeWithFallback(
+ s"""\"$jsonName\"""",
+ parser = DataType.fromDDL,
+ fallbackParser = DataType.fromJson) === TimestampLTZNanosType(p))
+ }
+ }
+ }
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
index 758e7eaf2158..636d16c78615 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
@@ -22,7 +22,7 @@ import java.util.concurrent.TimeUnit
import scala.language.implicitConversions
-import org.apache.spark.SparkThrowable
+import org.apache.spark.{SparkException, SparkThrowable}
import org.apache.spark.sql.catalyst.FunctionIdentifier
import org.apache.spark.sql.catalyst.analysis.{UnresolvedAttribute, _}
import org.apache.spark.sql.catalyst.expressions._
@@ -315,6 +315,36 @@ class ExpressionParserSuite extends AnalysisTest {
assertEqual("cast(cast(a as int) as long)",
$"a".cast(IntegerType).cast(LongType))
}
+ test("SPARK-57164: CAST / TRY_CAST to nanos timestamp types") {
+ import
org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" ->
TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ assertEqual(s"cast(a as $spelling)", $"a".cast(expected))
+ assertEqual(s"try_cast(a as $spelling)",
+ Cast($"a", expected, evalMode = EvalMode.TRY))
+ }
+ }
+ }
+ // With the preview flag off, a nanos data type in a cast target is
rejected.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ Seq("cast(a as TIMESTAMP_NTZ(9))", "try_cast(a as
TIMESTAMP_LTZ(7))").foreach { sql =>
+ checkError(
+ exception =
intercept[SparkException](defaultParser.parseExpression(sql)),
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+ }
+
test("function expressions") {
assertEqual("foo()", $"foo".function())
assertEqual("foo.bar()",
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
index 0334e90af68f..ff395ea70566 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
@@ -1556,6 +1556,16 @@ class DataTypeSuite extends SparkFunSuite with SQLHelper
{
assert(DataType.fromJson(arrayOfNanos.json) === arrayOfNanos)
val mapOfNanos = MapType(StringType, TimestampNTZNanosType(7),
valueContainsNull = true)
assert(DataType.fromJson(mapOfNanos.json) === mapOfNanos)
+
+ // Family B agrees with Family A: a nanos type's typeName re-parses (as
a JSON name) to
+ // the same type. For atomic types `json` is exactly the quoted
`typeName`, so this pins
+ // the type-name spelling that Family A produces against Family B's JSON
parser.
+ variants.foreach { case (_, _, factory) =>
+ TimestampLTZNanosType.MIN_PRECISION to
TimestampLTZNanosType.MAX_PRECISION foreach { n =>
+ val t = factory(n)
+ assert(DataType.fromJson("\"" + t.typeName + "\"") === t)
+ }
+ }
}
// Bare names without parens still map to the legacy single-precision
types, regardless
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
index 919a1a258d9b..379b7320a299 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
@@ -31,6 +31,7 @@ import org.apache.spark.sql.UpdateFieldsBenchmark._
import org.apache.spark.sql.catalyst.expressions.{InSet, Literal,
NamedExpression}
import
org.apache.spark.sql.catalyst.util.DateTimeTestUtils.{outstandingTimezonesIds,
outstandingZoneIds}
import org.apache.spark.sql.catalyst.util.DateTimeUtils
+import
org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.execution.ProjectExec
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
@@ -3164,4 +3165,38 @@ class ColumnExpressionSuite extends SharedSparkSession {
Seq(20, 40, 60).toDF()
)
}
+
+ test("SPARK-57164: Column.cast/try_cast(String) and sessionState parser for
nanos types") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" ->
TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ // Column.cast(String) / Column.try_cast(String) parse the type at
call time.
+ assert($"id".cast(spelling).expr.dataType === expected)
+ assert($"id".try_cast(spelling).expr.dataType === expected)
+ // The programmatic catalog-string entry point.
+ assert(spark.sessionState.sqlParser.parseDataType(spelling) ===
expected)
+ }
+ }
+ }
+ // With the preview flag off, the string overloads reject the nanos
spelling at call time.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ Seq[() => Any](
+ () => $"id".cast("TIMESTAMP_NTZ(9)"),
+ () => $"id".try_cast("TIMESTAMP_LTZ(7)"),
+ () =>
spark.sessionState.sqlParser.parseDataType("TIMESTAMP_NTZ(9)")).foreach { f =>
+ checkError(
+ exception = intercept[SparkException](f()),
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
index 70dbb7c8ce07..a70ef16bdf71 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
@@ -26,6 +26,7 @@ import scala.jdk.CollectionConverters._
import org.apache.spark.{SparkException, SparkRuntimeException,
SparkUnsupportedOperationException, SparkUpgradeException}
+import
org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.errors.DataTypeErrors.toSQLType
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
@@ -47,6 +48,31 @@ class CsvFunctionsSuite extends SharedSparkSession {
Row(Row(1)) :: Nil)
}
+ test("SPARK-57164: from_csv with a nanos timestamp DDL schema string") {
+ val df = Seq("2020-01-01T00:00:00.123456789").toDF("value")
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" ->
TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ val parsed = df.select(
+ from_csv($"value", lit(s"c $spelling"), Map.empty[String,
String].asJava).as("v"))
+ // The schema string resolves to the nanos type ...
+
assert(parsed.schema("v").dataType.asInstanceOf[StructType]("c").dataType ===
expected)
+ // ... but the CSV datasource does not support nanosecond
timestamps yet, so the
+ // value converter rejects it at execution.
+ checkError(
+ exception =
intercept[SparkUnsupportedOperationException](parsed.collect()),
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = Map("typeName" -> toSQLType(expected)))
+ }
+ }
+ }
+ }
+
test("from_csv with non struct schema") {
checkError(
exception = intercept[AnalysisException] {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
index 6b50dcd0871b..8857a9977b57 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
@@ -23,10 +23,12 @@ import java.util.Locale
import scala.jdk.CollectionConverters._
-import org.apache.spark.{SparkException, SparkRuntimeException}
+import org.apache.spark.{SparkException, SparkRuntimeException,
SparkUnsupportedOperationException}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.Cast._
import org.apache.spark.sql.catalyst.expressions.Literal
+import
org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
+import org.apache.spark.sql.errors.DataTypeErrors.toSQLType
import org.apache.spark.sql.execution.WholeStageCodegenExec
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
@@ -365,6 +367,31 @@ class JsonFunctionsSuite extends SharedSparkSession {
Row(Row(1, "haa")) :: Nil)
}
+ test("SPARK-57164: from_json with a nanos timestamp DDL schema string") {
+ val df = Seq("""{"c": "2020-01-01T00:00:00.123456789"}""").toDF("value")
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" ->
TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ val parsed = df.select(
+ from_json($"value", s"c $spelling", Map.empty[String,
String]).as("v"))
+ // The schema string resolves to the nanos type ...
+
assert(parsed.schema("v").dataType.asInstanceOf[StructType]("c").dataType ===
expected)
+ // ... but the JSON datasource does not support nanosecond
timestamps yet, so the
+ // value converter rejects it at execution.
+ checkError(
+ exception =
intercept[SparkUnsupportedOperationException](parsed.collect()),
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = Map("typeName" -> toSQLType(expected)))
+ }
+ }
+ }
+ }
+
test("from_json with NULL in options map values") {
val df = Seq("""{"str": "World"}""").toDS()
val schema = "str STRING"
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/XmlFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/XmlFunctionsSuite.scala
index d07758721fca..f87bd2df5164 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/XmlFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/XmlFunctionsSuite.scala
@@ -22,6 +22,8 @@ import java.util.Locale
import scala.jdk.CollectionConverters._
+import org.apache.spark.SparkException
+import
org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.execution.WholeStageCodegenExec
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
@@ -40,6 +42,33 @@ class XmlFunctionsSuite extends SharedSparkSession {
Row(Row(1)) :: Nil)
}
+ test("SPARK-57164: from_xml with a nanos timestamp DDL schema string") {
+ val df =
Seq("""<ROW><c>2020-01-01T00:00:00.123456789</c></ROW>""").toDF("value")
+ // FAILFAST so the value-converter rejection propagates instead of
becoming a corrupt record.
+ val options = Map("mode" -> "FAILFAST").asJava
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" ->
TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ val parsed = df.select(from_xml($"value", s"c $spelling",
options).as("v"))
+ // The schema string resolves to the nanos type ...
+
assert(parsed.schema("v").dataType.asInstanceOf[StructType]("c").dataType ===
expected)
+ // ... but the XML datasource does not support nanosecond
timestamps yet, so the
+ // value converter rejects it at execution (surfaced as a
malformed record in
+ // FAILFAST mode).
+ checkError(
+ exception = intercept[SparkException](parsed.collect()),
+ condition = "MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION",
+ parameters = Map("badRecord" -> "[null]", "failFastMode" ->
"FAILFAST"))
+ }
+ }
+ }
+ }
+
test("SPARK-48300: from_xml - Codegen Support") {
withTempView("XmlToStructsTable") {
val dataDF = Seq("""<ROW><a>1</a></ROW>""").toDF("value")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala
index 2a186a9296f4..a1dfefe3fb89 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala
@@ -29,8 +29,10 @@ import org.mockito.ArgumentMatchers.{any, eq => meq}
import org.mockito.Mockito._
import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkException
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
+import
org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.classic.ClassicConversions.castToImpl
import org.apache.spark.sql.classic.Dataset.ofRows
import org.apache.spark.sql.execution.datasources.DataSourceUtils
@@ -163,6 +165,41 @@ class DataStreamReaderWriterSuite extends StreamTest with
BeforeAndAfter {
.stop()
}
+ test("SPARK-57164: DataStreamReader.schema(String) parses nanos timestamp
types") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" ->
TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ LastOptions.clear()
+ spark.readStream
+ .format("org.apache.spark.sql.streaming.test")
+ .schema(s"c $spelling")
+ .load()
+ val captured = LastOptions.schema.get
+ assert(captured("c").dataType === expected)
+ }
+ }
+ }
+ // With the preview flag off, schema(String) rejects the nanos spelling at
parse time.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ checkError(
+ exception = intercept[SparkException] {
+ spark.readStream
+ .format("org.apache.spark.sql.streaming.test")
+ .schema("c TIMESTAMP_NTZ(9)")
+ },
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+
test("options") {
val map = new java.util.HashMap[String, String]
map.put("opt3", "3")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
index 37dd0fc3871e..cf05be197e24 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
@@ -32,13 +32,14 @@ import
org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName
import org.apache.parquet.schema.Type.Repetition
import org.scalatest.BeforeAndAfter
-import org.apache.spark.{SparkContext, SparkIllegalArgumentException,
TestUtils}
+import org.apache.spark.{SparkContext, SparkException,
SparkIllegalArgumentException, TestUtils}
import org.apache.spark.internal.io.FileCommitProtocol.TaskCommitMessage
import org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
import org.apache.spark.scheduler.{SparkListener, SparkListenerJobStart}
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.plans.logical.{AppendData, LogicalPlan,
OverwriteByExpression}
+import
org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.execution.QueryExecution
import org.apache.spark.sql.execution.command.DataWritingCommandExec
import org.apache.spark.sql.execution.datasources.{DataSourceUtils,
HadoopFsRelation, LogicalRelation}
@@ -197,6 +198,39 @@ class DataFrameReaderWriterSuite extends
SharedSparkSession with BeforeAndAfter
.save()
}
+ test("SPARK-57164: DataFrameReader.schema(String) parses nanos timestamp
types") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" ->
TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ LastOptions.clear()
+ spark.read
+ .format("org.apache.spark.sql.test")
+ .schema(s"c $spelling")
+ .load()
+ val captured = LastOptions.schema.get
+ assert(captured("c").dataType === expected)
+ }
+ }
+ }
+ // With the preview flag off, schema(String) rejects the nanos spelling at
parse time.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ checkError(
+ exception = intercept[SparkException] {
+ spark.read.format("org.apache.spark.sql.test").schema("c
TIMESTAMP_NTZ(9)")
+ },
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+
test("options") {
val map = new java.util.HashMap[String, String]
map.put("opt3", "3")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
