This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new fd073191106a [SPARK-57551][SQL] Extend the TIME data type precision to
nanoseconds (up to 9)
fd073191106a is described below
commit fd073191106aa57a4437f833b3ab47c042924fa0
Author: Maxim Gekk <[email protected]>
AuthorDate: Mon Jun 22 13:47:10 2026 +0200
[SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up
to 9)
### What changes were proposed in this pull request?
This PR extends the fractional-seconds precision of the `TIME` data type
from a maximum of 6 (microseconds) to 9 (nanoseconds), so `TIME(p)` accepts `0
<= p <= 9`. The internal storage is already nanoseconds-since-midnight (`Long`,
`TimeType.NANOS_PRECISION = 9` already exists), so this lifts the cap and
closes the two remaining micros-only code paths:
- `TimeType.MAX_PRECISION` is raised from 6 to 9 (`DEFAULT_PRECISION` stays
6). The `UNSUPPORTED_TIME_PRECISION` error message and related
scaladoc/comments are updated to `[0, 9]`.
- `SparkDateTimeUtils.stringToTime` now keeps the sub-microsecond digits
(7-9), mirroring the timestamp parser, and `CAST(<string> AS TIME(p))`
truncates the parsed value to the target precision (interpreted and codegen
paths).
- `CurrentTime` accepts precisions up to `MAX_PRECISION`.
- Parquet I/O emits/reads `TIME(NANOS)` for precision 7..9 (and keeps
`TIME(MICROS)` for 0..6) across `TimeTypeParquetOps`, `ParquetSchemaConverter`,
the vectorized `ParquetVectorUpdaterFactory`, and the legacy row/write
fallbacks. On read, the value is truncated to the requested precision in both
the vectorized and the row-based readers, so a higher-precision file read with
a lower precision (e.g. a `TIME(NANOS)` file read as `TIME(7)`) gives the same
result either way.
- ORC stores the raw nanosecond `Long` with the catalyst type name
preserved, so it round-trips 7..9 losslessly without production changes. Avro
encodes `TIME` as the `time-micros` logical type (SPARK-57581) and has no
`time-nanos` type yet (upstream AVRO-4043), so a `TIME(7-9)` value written to
Avro is truncated to microseconds (the declared precision metadata is still
preserved via `spark.sql.catalyst.type`).
Dictionary-encoded `TIME(NANOS)` columns disable lazy dictionary decoding
so reads still pass through the truncating path (as `TIME(MICROS)` already
does); keeping lazy decoding for `TIME` is left to SPARK-57583.
Casts that were already nanosecond-aware (`TIME(p1) -> TIME(p2)`, `TIME ->
DECIMAL`, `TIME -> integral`, `TIME -> STRING`) work for 7..9 once the cap is
lifted.
This PR regenerates `time.sql.out` and `cast.sql.out` for pre-existing
queries whose output changes once 7-9 fractional digits are preserved (e.g.
`CAST(time '23:59:59.999999999' AS decimal(...))`, `time_trunc(...,
time'...123456789')`). That is an unavoidable golden-file update for
already-present queries; the broader golden-file parity effort (new try_cast /
datetime parsing-formatting / postgreSQL coverage) remains out of scope and is
tracked by SPARK-57563.
Out of scope (tracked separately): casts to/from TIMESTAMP types
(SPARK-57552 / SPARK-57554), lazy Parquet dictionary decoding for `TIME`
(SPARK-57583), and unit-correct Avro encoding of `TIME(7-9)` (blocked upstream
by AVRO-4043; the micros encoding for precision 0-6 landed in SPARK-57581).
### Why are the changes needed?
ANSI SQL (ISO/IEC 9075-2, 6.1 `<data type>`) makes the maximum `<time
precision>` implementation-defined with the sole constraint that it is not less
than 6, and requires the maximum of `<time precision>` and `<timestamp
precision>` to be the same implementation-defined value. Spark already supports
nanosecond timestamps, so to stay ANSI-consistent `TIME` must reach precision 9
in lockstep.
### Does this PR introduce _any_ user-facing change?
Yes. `TIME(7)`, `TIME(8)`, and `TIME(9)` can now be declared, parsed, used
as literals, and round-tripped losslessly through Parquet and ORC. (Avro stores
`TIME` as `time-micros`, so `TIME(7-9)` is truncated to microseconds over Avro
until Avro adds a `time-nanos` logical type, AVRO-4043.) Previously these
precisions raised `UNSUPPORTED_TIME_PRECISION`. The default precision of `TIME`
is unchanged (6).
### How was this patch tested?
Extended existing TIME suites to cover precision 7..9
(`DataTypeParserSuite`, `DataTypeSuite`, `TimeExpressionsSuite`,
`CastWithAnsiOn/OffSuite`, `TimeFormatterSuite`, `DateTimeUtilsSuite`,
CSV/JSON/XML expression and function suites, `TimeTypeParquetOpsSuite`,
`ParquetIOSuite`, `OrcQuerySuite`, `AvroSuite`/`AvroFunctionsSuite`,
`PartitionedWriteSuite`, `RowJsonSuite`, `SparkConnectPlannerSuite`) and added
nanosecond Parquet read and round-trip tests for both readers, plus a test that
[...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
Closes #56622 from MaxGekk/time-sub-micro-precision.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 8 +-
.../apache/spark/sql/avro/AvroFunctionsSuite.scala | 17 +++--
.../org/apache/spark/sql/avro/AvroSuite.scala | 27 +++++++
.../sql/catalyst/util/SparkDateTimeUtils.scala | 20 +++--
.../spark/sql/catalyst/util/TimeFormatter.scala | 2 +-
.../spark/sql/errors/QueryParsingErrors.scala | 9 +++
.../org/apache/spark/sql/types/TimeType.scala | 10 +--
.../spark/sql/catalyst/expressions/Cast.scala | 12 ++-
.../sql/catalyst/expressions/timeExpressions.scala | 6 +-
.../spark/sql/catalyst/parser/AstBuilder.scala | 14 +++-
.../spark/sql/catalyst/types/ops/TimeTypeOps.scala | 2 +-
.../org/apache/spark/sql/RandomDataGenerator.scala | 10 ++-
.../scala/org/apache/spark/sql/RowJsonSuite.scala | 4 +-
.../sql/catalyst/expressions/CastSuiteBase.scala | 28 +++++++
.../catalyst/expressions/CsvExpressionsSuite.scala | 5 +-
.../expressions/JsonExpressionsSuite.scala | 5 +-
.../catalyst/expressions/LiteralGenerator.scala | 16 ++--
.../expressions/TimeExpressionsSuite.scala | 14 +++-
.../catalyst/expressions/XmlExpressionsSuite.scala | 5 +-
.../sql/catalyst/parser/DataTypeParserSuite.scala | 12 ++-
.../catalyst/parser/ExpressionParserSuite.scala | 23 ++++++
.../sql/catalyst/util/DateTimeTestUtils.scala | 11 ++-
.../sql/catalyst/util/DateTimeUtilsSuite.scala | 20 +++++
.../sql/catalyst/util/TimeFormatterSuite.scala | 34 ++++++++-
.../org/apache/spark/sql/types/DataTypeSuite.scala | 6 +-
.../parquet/ParquetVectorUpdaterFactory.java | 56 ++++++++++++--
.../parquet/VectorizedColumnReader.java | 6 +-
.../datasources/parquet/ParquetRowConverter.scala | 17 +++--
.../parquet/ParquetSchemaConverter.scala | 9 ++-
.../datasources/parquet/ParquetWriteSupport.scala | 5 ++
.../parquet/types/ops/TimeTypeParquetOps.scala | 87 ++++++++++++++--------
.../sql-tests/analyzer-results/cast.sql.out | 6 +-
.../analyzer-results/nonansi/cast.sql.out | 6 +-
.../sql-tests/analyzer-results/time.sql.out | 60 +++++++--------
.../test/resources/sql-tests/results/cast.sql.out | 12 +--
.../sql-tests/results/nonansi/cast.sql.out | 12 +--
.../test/resources/sql-tests/results/time.sql.out | 60 +++++++--------
.../org/apache/spark/sql/CsvFunctionsSuite.scala | 5 +-
.../org/apache/spark/sql/JsonFunctionsSuite.scala | 5 +-
.../apache/spark/sql/TimeFunctionsSuiteBase.scala | 2 +-
.../org/apache/spark/sql/XmlFunctionsSuite.scala | 5 +-
.../execution/datasources/orc/OrcQuerySuite.scala | 7 +-
.../datasources/parquet/ParquetIOSuite.scala | 82 ++++++++++++++++++++
.../types/ops/TimeTypeParquetOpsSuite.scala | 47 +++++++-----
.../spark/sql/sources/PartitionedWriteSuite.scala | 5 +-
45 files changed, 605 insertions(+), 209 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index 8e7125a07c1e..13130579c6a1 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -5028,6 +5028,12 @@
],
"sqlState" : "22009"
},
+ "INVALID_TIME_LITERAL_PRECISION" : {
+ "message" : [
+ "The time literal <value> has more than 9 fractional-second digits. The
maximum supported fractional-second precision of a time literal is 9
(nanoseconds)."
+ ],
+ "sqlState" : "22023"
+ },
"INVALID_TIME_TRAVEL_SPEC" : {
"message" : [
"Cannot specify both version and timestamp when time travelling the
table."
@@ -8857,7 +8863,7 @@
},
"UNSUPPORTED_TIME_PRECISION" : {
"message" : [
- "The seconds precision <precision> of the TIME data type is out of the
supported range [0, 6]."
+ "The seconds precision <precision> of the TIME data type is out of the
supported range [0, 9]."
],
"sqlState" : "0A001"
},
diff --git
a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala
b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala
index 4da104fa4647..bcc104aa3e82 100644
---
a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala
+++
b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala
@@ -668,15 +668,18 @@ class AvroFunctionsSuite extends SharedSparkSession {
}
test("roundtrip in to_avro and from_avro - TIME type with different
precisions") {
+ // Avro stores TIME as the time-micros logical type, so this lossless
round-trip is limited to
+ // precision 0-6. TIME(7-9) truncates to microseconds over Avro (no
time-nanos logical type,
+ // upstream AVRO-4043); that behavior is covered in AvroSuite.
val df = spark.sql("""
SELECT
- TIME'12:34:56' as time_p0,
- TIME'12:34:56.1' as time_p1,
- TIME'12:34:56.12' as time_p2,
- TIME'12:34:56.123' as time_p3,
- TIME'12:34:56.1234' as time_p4,
- TIME'12:34:56.12345' as time_p5,
- TIME'12:34:56.123456' as time_p6
+ CAST(TIME'12:34:56' AS TIME(0)) as time_p0,
+ CAST(TIME'12:34:56.1' AS TIME(1)) as time_p1,
+ CAST(TIME'12:34:56.12' AS TIME(2)) as time_p2,
+ CAST(TIME'12:34:56.123' AS TIME(3)) as time_p3,
+ CAST(TIME'12:34:56.1234' AS TIME(4)) as time_p4,
+ CAST(TIME'12:34:56.12345' AS TIME(5)) as time_p5,
+ CAST(TIME'12:34:56.123456' AS TIME(6)) as time_p6
""")
val precisions = Seq(0, 1, 2, 3, 4, 5, 6)
diff --git
a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index 98b240cc56b3..588d7e26206f 100644
--- a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -3322,6 +3322,33 @@ abstract class AvroSuite
}
}
+ test("SPARK-57551: TIME(7-9) is truncated to microseconds when written to
Avro") {
+ // Avro has no time-nanos logical type (upstream AVRO-4043), so TIME is
stored as time-micros.
+ // Writing a TIME(7-9) value therefore drops the sub-microsecond digits,
while the column's
+ // precision metadata (time(p)) is still preserved via the
spark.sql.catalyst.type property.
+ withTempPath { dir =>
+ val df = spark.sql("""
+ SELECT
+ CAST(TIME '12:34:56.1234567' AS TIME(7)) as time_p7,
+ CAST(TIME '12:34:56.12345678' AS TIME(8)) as time_p8,
+ CAST(TIME '12:34:56.123456789' AS TIME(9)) as time_p9
+ """)
+
+ df.write.format("avro").save(dir.toString)
+ val readDf = spark.read.format("avro").load(dir.toString)
+
+ // The declared precision is preserved.
+ Seq(7, 8, 9).foreach { p =>
+ assert(readDf.schema(s"time_p$p").dataType == TimeType(p),
+ s"Precision $p should be preserved")
+ }
+
+ // The value reads back truncated to microsecond resolution (.123456789
-> .123456).
+ val micros = java.time.LocalTime.of(12, 34, 56, 123456000)
+ checkAnswer(readDf, Row(micros, micros, micros))
+ }
+ }
+
test("SPARK-57581: TIME is written as unit-correct time-micros for external
readers") {
// Expected microseconds-since-midnight for TIME'12:34:56.123456'
truncated to each precision.
val baseSeconds = (12 * 3600 + 34 * 60 + 56).toLong
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala
index 2b992ba53fe8..430b04d8f3df 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala
@@ -169,7 +169,7 @@ trait SparkDateTimeUtils {
* @param nanos
* The original time in nanoseconds.
* @param p
- * The fractional second precision (range 0 to 6).
+ * The fractional second precision (range 0 to 9).
* @return
* The truncated nanosecond value, preserving only `p` fractional digits.
*/
@@ -178,11 +178,16 @@ trait SparkDateTimeUtils {
TimeType.MIN_PRECISION <= p && p <= TimeType.MAX_PRECISION,
s"Fractional second precision $p out" +
s" of range [${TimeType.MIN_PRECISION}..${TimeType.MAX_PRECISION}].")
- val scale = TimeType.NANOS_PRECISION - p
- val factor = math.pow(10, scale).toLong
+ val factor = timeTruncationFactors(TimeType.NANOS_PRECISION - p)
(nanos / factor) * factor
}
+ // Precomputed 10^k for k in [0, NANOS_PRECISION], indexed by the truncation
scale
+ // (NANOS_PRECISION - p). `truncateTimeToPrecision` runs per value on hot
read/cast paths, so the
+ // factor is looked up here instead of recomputed with `math.pow` on every
call.
+ private val timeTruncationFactors: Array[Long] =
+ (0 to TimeType.NANOS_PRECISION).map(k => math.pow(10, k).toLong).toArray
+
/**
* Converts the timestamp `micros` from one timezone to another.
*
@@ -1080,8 +1085,10 @@ trait SparkDateTimeUtils {
return None
}
- // Unpack the segments.
- var (hr, min, sec, ms) = (segments(3), segments(4), segments(5),
segments(6))
+ // Unpack the segments. `segments(6)` holds microseconds and
`segments(9)` holds the
+ // sub-microsecond nanosecond remainder (digits 7-9), in [0, 999].
+ var (hr, min, sec, ms, subMicroNanos) =
+ (segments(3), segments(4), segments(5), segments(6), segments(9))
// Handle AM/PM conversion in separate cases.
if (!hasSuffix) {
@@ -1108,7 +1115,8 @@ trait SparkDateTimeUtils {
}
}
- val localTime = LocalTime.of(hr, min, sec,
MICROSECONDS.toNanos(ms).toInt)
+ val nanoOfSecond = (MICROSECONDS.toNanos(ms) + subMicroNanos).toInt
+ val localTime = LocalTime.of(hr, min, sec, nanoOfSecond)
Some(localTimeToNanos(localTime))
} catch {
case NonFatal(_) => None
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimeFormatter.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimeFormatter.scala
index d0438c6ff1b4..a8c5225353d7 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimeFormatter.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimeFormatter.scala
@@ -68,7 +68,7 @@ class Iso8601TimeFormatter(pattern: String, locale: Locale,
isParsing: Boolean)
/**
* The formatter parses/formats times according to the pattern
`HH:mm:ss.[..fff..]` where
- * `[..fff..]` is a fraction of second up to microsecond resolution. The
formatter does not output
+ * `[..fff..]` is a fraction of second up to nanosecond resolution. The
formatter does not output
* trailing zeros in the fraction. For example, the time `15:00:01.123400` is
formatted as the
* string `15:00:01.1234`.
*/
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala
b/sql/api/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala
index 1cc050f488f6..558bda49a302 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala
@@ -369,6 +369,15 @@ private[sql] object QueryParsingErrors extends
DataTypeErrorsBase {
ctx)
}
+ def timeLiteralPrecisionExceedsMaxError(
+ value: String,
+ ctx: TypeConstructorContext): Throwable = {
+ new ParseException(
+ errorClass = "INVALID_TIME_LITERAL_PRECISION",
+ messageParameters = Map("value" -> toSQLValue(value)),
+ ctx)
+ }
+
def literalValueTypeUnsupportedError(
unsupportedType: String,
supportedTypes: Seq[String],
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/TimeType.scala
b/sql/api/src/main/scala/org/apache/spark/sql/types/TimeType.scala
index 135ad278438e..b2cc29f5379e 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/TimeType.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/TimeType.scala
@@ -21,12 +21,12 @@ import org.apache.spark.annotation.Unstable
import org.apache.spark.sql.errors.DataTypeErrors
/**
- * The time type represents a time value with fields hour, minute, second, up
to microseconds. The
- * range of times supported is 00:00:00.000000 to 23:59:59.999999.
+ * The time type represents a time value with fields hour, minute, second, up
to nanoseconds. The
+ * range of times supported is 00:00:00.000000000 to 23:59:59.999999999.
*
* @param precision
* The time fractional seconds precision which indicates the number of
decimal digits maintained
- * following the decimal point in the seconds value. The supported range is
[0, 6].
+ * following the decimal point in the seconds value. The supported range is
[0, 9].
*
* @since 4.1.0
*/
@@ -50,9 +50,9 @@ case class TimeType(precision: Int) extends AnyTimeType {
object TimeType {
val MIN_PRECISION: Int = 0
val MICROS_PRECISION: Int = 6
- val MAX_PRECISION: Int = MICROS_PRECISION
- val DEFAULT_PRECISION: Int = MICROS_PRECISION
val NANOS_PRECISION: Int = 9
+ val MAX_PRECISION: Int = NANOS_PRECISION
+ val DEFAULT_PRECISION: Int = MICROS_PRECISION
def apply(): TimeType = new TimeType(DEFAULT_PRECISION)
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
index 506babb08f34..62bbd65863e8 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
@@ -965,9 +965,11 @@ case class Cast(
private[this] def castToTime(from: DataType, to: TimeType): Any => Any =
from match {
case _: StringType =>
if (ansiEnabled) {
- buildCast[UTF8String](_, s => DateTimeUtils.stringToTimeAnsi(s,
getContextOrNull()))
+ buildCast[UTF8String](_, s => DateTimeUtils.truncateTimeToPrecision(
+ DateTimeUtils.stringToTimeAnsi(s, getContextOrNull()), to.precision))
} else {
- buildCast[UTF8String](_, s => DateTimeUtils.stringToTime(s).orNull)
+ buildCast[UTF8String](_, s => DateTimeUtils.stringToTime(s)
+ .map(DateTimeUtils.truncateTimeToPrecision(_, to.precision)).orNull)
}
case _: TimeType =>
buildCast[Long](_, nanos => DateTimeUtils.truncateTimeToPrecision(nanos,
to.precision))
@@ -1670,13 +1672,15 @@ case class Cast(
if (ansiEnabled) {
val errorContext = getContextOrNullCode(ctx)
code"""
- $evPrim = $dateTimeUtilsCls.stringToTimeAnsi($c, $errorContext);
+ $evPrim = $dateTimeUtilsCls.truncateTimeToPrecision(
+ $dateTimeUtilsCls.stringToTimeAnsi($c, $errorContext),
${to.precision});
"""
} else {
code"""
scala.Option<Long> $longOpt = $dateTimeUtilsCls.stringToTime($c);
if ($longOpt.isDefined()) {
- $evPrim = ((Long) $longOpt.get()).longValue();
+ $evPrim = $dateTimeUtilsCls.truncateTimeToPrecision(
+ ((Long) $longOpt.get()).longValue(), ${to.precision});
} else {
$evNull = true;
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala
index 85afdf72eecc..1c9d6b335e8d 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala
@@ -467,7 +467,7 @@ object SecondExpressionBuilder extends ExpressionBuilder {
""",
arguments = """
Arguments:
- * precision - An optional integer literal in the range [0..6],
indicating how many
+ * precision - An optional integer literal in the range [0..9],
indicating how many
fractional digits of seconds to include. If omitted, the
default is 6.
""",
examples = """
@@ -531,12 +531,12 @@ case class CurrentTime(
precisionValue match {
case n: Number =>
val p = n.intValue()
- if (p < TimeType.MIN_PRECISION || p > TimeType.MICROS_PRECISION) {
+ if (p < TimeType.MIN_PRECISION || p > TimeType.MAX_PRECISION) {
return DataTypeMismatch(
errorSubClass = "VALUE_OUT_OF_RANGE",
messageParameters = Map(
"exprName" -> toSQLId("precision"),
- "valueRange" -> s"[${TimeType.MIN_PRECISION},
${TimeType.MICROS_PRECISION}]",
+ "valueRange" -> s"[${TimeType.MIN_PRECISION},
${TimeType.MAX_PRECISION}]",
"currentValue" -> toSQLValue(p, IntegerType)
)
)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
index 4dc63760e6a2..43e094b5ccec 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
@@ -4136,7 +4136,19 @@ class AstBuilder extends DataTypeAstBuilder
val zoneId = getZoneId(conf.sessionLocalTimeZone)
val specialDate = convertSpecialDate(value, zoneId).map(Literal(_,
DateType))
specialDate.getOrElse(toLiteral(stringToDate, DateType))
- case TIME => toLiteral(stringToTime, TimeType())
+ case TIME =>
+ // ANSI SQL (ISO/IEC 9075-2, Subclause 5.3, Syntax Rule 26): the
fractional-seconds
+ // precision of a time literal is the number of digits in its seconds
fraction. A literal
+ // with 7-9 fractional digits becomes a nanosecond-precision literal;
<= 6 digits keep the
+ // default microsecond precision; more than 9 digits is rejected.
+ val p = fractionalSecondsDigits(value)
+ if (p > TimeType.MAX_PRECISION) {
+ throw QueryParsingErrors.timeLiteralPrecisionExceedsMaxError(value,
ctx)
+ } else if (p > TimeType.MICROS_PRECISION) {
+ toLiteral(stringToTime, TimeType(p))
+ } else {
+ toLiteral(stringToTime, TimeType())
+ }
case TIMESTAMP_NTZ =>
nanosLiteralOpt(constructTimestampNTZNanosLiteral).getOrElse {
convertSpecialTimestampNTZ(value,
getZoneId(conf.sessionLocalTimeZone))
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimeTypeOps.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimeTypeOps.scala
index d1700aad05cf..3ff1d6c9165a 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimeTypeOps.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimeTypeOps.scala
@@ -49,7 +49,7 @@ import org.apache.spark.sql.types.{ObjectType, TimeType}
* - Values stored as Long nanoseconds since midnight
* - Range: 0 to 86,399,999,999,999
* - External type: java.time.LocalTime
- * - Precision (0-6) affects display only, not storage
+ * - Precision (0-9) affects display only, not storage
*
* @param t
* The TimeType with precision information
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala
index 289b2ff6851f..1390155fc7b6 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala
@@ -318,15 +318,19 @@ object RandomDataGenerator {
.map(s => TimestampNanosTestUtils.parseSpecialNanosLTZ(s,
ZoneId.systemDefault()))
.map(i => Instant.ofEpochSecond(i.getEpochSecond,
truncate(i.getNano).toLong))
)
- case _: TimeType =>
+ case t: TimeType =>
val specialTimes = Seq(
"00:00:00",
- "23:59:59.999999"
+ "23:59:59.999999",
+ "23:59:59.999999999"
)
randomNumeric[LocalTime](
rand,
(rand: Random) => {
- DateTimeUtils.nanosToLocalTime(rand.between(0, 24 * 60 * 60 * 1000
* 1000L) * 1000L)
+ // The full valid range is [0, 86_399_999_999_999] nanoseconds
since midnight.
+ val nanos = DateTimeUtils.truncateTimeToPrecision(
+ rand.between(0L, 24 * 60 * 60 * 1000 * 1000 * 1000L),
t.precision)
+ DateTimeUtils.nanosToLocalTime(nanos)
},
specialTimes.map(LocalTime.parse)
)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/RowJsonSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/RowJsonSuite.scala
index 31d967f3da37..6b228848feaf 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/RowJsonSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/RowJsonSuite.scala
@@ -134,11 +134,13 @@ class RowJsonSuite extends SparkFunSuite with SQLHelper {
test("SPARK-57338: TIME column renders the external LocalTime in JSON") {
assert(timeRowJson(LocalTime.of(12, 13, 14), TimeType.MICROS_PRECISION) ===
JObject("a" -> JString("12:13:14")))
- // The fraction is rendered up to microsecond resolution with trailing
zeros trimmed.
+ // The fraction is rendered up to nanosecond resolution with trailing
zeros trimmed.
assert(timeRowJson(LocalTime.of(1, 2, 3, 123456000),
TimeType.MICROS_PRECISION) ===
JObject("a" -> JString("01:02:03.123456")))
assert(timeRowJson(LocalTime.of(10, 30, 0, 100000000),
TimeType.MICROS_PRECISION) ===
JObject("a" -> JString("10:30:00.1")))
+ assert(timeRowJson(LocalTime.of(1, 2, 3, 123456789),
TimeType.NANOS_PRECISION) ===
+ JObject("a" -> JString("01:02:03.123456789")))
assert(timeRowJson(LocalTime.MIDNIGHT, TimeType.MICROS_PRECISION) ===
JObject("a" -> JString("00:00:00")))
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
index 980c68f65f20..122554a18886 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
@@ -2328,6 +2328,16 @@ abstract class CastSuiteBase extends SparkFunSuite with
ExpressionEvalHelper {
TimeType(5)), localTime(23, 59, 0, 999990))
checkEvaluation(cast(Literal.create("23:59:59.000001 "),
TimeType(6)), localTime(23, 59, 59, 1))
+ // Nanosecond precisions 7, 8, 9.
+ checkEvaluation(cast(Literal.create("01:02:03.1234567"),
+ TimeType(7)), localTime(1, 2, 3, 123456, 700))
+ checkEvaluation(cast(Literal.create("01:02:03.12345678"),
+ TimeType(8)), localTime(1, 2, 3, 123456, 780))
+ checkEvaluation(cast(Literal.create("01:02:03.123456789"),
+ TimeType(9)), localTime(1, 2, 3, 123456, 789))
+ // More fractional digits than the target precision are truncated.
+ checkEvaluation(cast(Literal.create("01:02:03.123456789"),
+ TimeType(7)), localTime(1, 2, 3, 123456, 700))
}
test("context independent foldable") {
@@ -2474,6 +2484,16 @@ abstract class CastSuiteBase extends SparkFunSuite with
ExpressionEvalHelper {
localTime(11, 58, 59, 123400))
checkEvaluation(cast(Literal(localTime(19, 2, 3, 765000), TimeType(3)),
TimeType(2)),
localTime(19, 2, 3, 760000))
+ // Nanosecond precisions 7, 8, 9.
+ checkEvaluation(cast(Literal(localTime(23, 59, 59, 999999, 999),
TimeType(9)), TimeType(9)),
+ localTime(23, 59, 59, 999999, 999))
+ checkEvaluation(cast(Literal(localTime(1, 2, 3, 123456, 789),
TimeType(9)), TimeType(8)),
+ localTime(1, 2, 3, 123456, 780))
+ checkEvaluation(cast(Literal(localTime(1, 2, 3, 123456, 789),
TimeType(9)), TimeType(7)),
+ localTime(1, 2, 3, 123456, 700))
+ // Truncate nanosecond value down to the microsecond precision.
+ checkEvaluation(cast(Literal(localTime(1, 2, 3, 123456, 789),
TimeType(9)), TimeType(6)),
+ localTime(1, 2, 3, 123456))
for (sp <- TimeType.MIN_PRECISION to TimeType.MAX_PRECISION) {
for (tp <- TimeType.MIN_PRECISION to TimeType.MAX_PRECISION) {
@@ -2642,5 +2662,13 @@ abstract class CastSuiteBase extends SparkFunSuite with
ExpressionEvalHelper {
checkEvaluation(cast(oneTwoThreeTime5, LongType), 3723L)
checkEvaluation(cast(maxTime4, IntegerType), 86399)
checkEvaluation(cast(maxTime4, LongType), 86399L)
+
+ // Nanosecond precisions: sub-second fraction is truncated to whole
seconds.
+ val nanos9 = Literal.create(LocalTime.of(0, 0, 17, 999999999), TimeType(9))
+ checkEvaluation(cast(nanos9, IntegerType), 17)
+ checkEvaluation(cast(nanos9, LongType), 17L)
+ val maxNanos = Literal.create(LocalTime.of(23, 59, 59, 999999999),
TimeType(9))
+ checkEvaluation(cast(maxNanos, IntegerType), 86399)
+ checkEvaluation(cast(maxNanos, LongType), 86399L)
}
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CsvExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CsvExpressionsSuite.scala
index d168735d4045..631b08b5395f 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CsvExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CsvExpressionsSuite.scala
@@ -268,7 +268,10 @@ class CsvExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
(3, LocalTime.of(14, 30, 45, 123000000), "14:30:45.123"),
(4, LocalTime.of(14, 30, 45, 123400000), "14:30:45.1234"),
(5, LocalTime.of(14, 30, 45, 123450000), "14:30:45.12345"),
- (6, LocalTime.of(14, 30, 45, 123456000), "14:30:45.123456")
+ (6, LocalTime.of(14, 30, 45, 123456000), "14:30:45.123456"),
+ (7, LocalTime.of(14, 30, 45, 123456700), "14:30:45.1234567"),
+ (8, LocalTime.of(14, 30, 45, 123456780), "14:30:45.12345678"),
+ (9, LocalTime.of(14, 30, 45, 123456789), "14:30:45.123456789")
)
testData.foreach { case (precision, timeValue, timeStr) =>
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala
index ece8fbbd8265..7f6ae46be3a8 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala
@@ -916,7 +916,10 @@ class JsonExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
("14:30:45.123", 3),
("14:30:45.1234", 4),
("14:30:45.12345", 5),
- ("23:59:59.999999", 6)
+ ("23:59:59.999999", 6),
+ ("14:30:45.1234567", 7),
+ ("14:30:45.12345678", 8),
+ ("23:59:59.999999999", 9)
)
testCases.foreach { case (timeStr, precision) =>
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralGenerator.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralGenerator.scala
index 1e082e0f1163..d40c597404b0 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralGenerator.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralGenerator.scala
@@ -24,9 +24,9 @@ import java.util.concurrent.TimeUnit
import org.scalacheck.{Arbitrary, Gen}
import org.scalatest.Assertions._
-import
org.apache.spark.sql.catalyst.util.DateTimeConstants.{MICROS_PER_MILLIS,
MILLIS_PER_DAY, NANOS_PER_MICROS}
+import
org.apache.spark.sql.catalyst.util.DateTimeConstants.{MICROS_PER_MILLIS,
MILLIS_PER_DAY}
import org.apache.spark.sql.catalyst.util.DateTimeUtils
-import org.apache.spark.sql.catalyst.util.DateTimeUtils.{instantToMicros,
localTimeToNanos, nanosToMicros}
+import org.apache.spark.sql.catalyst.util.DateTimeUtils.{instantToMicros,
localTimeToNanos}
import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils
import org.apache.spark.sql.types._
import org.apache.spark.unsafe.types.{CalendarInterval, TimestampNanosVal}
@@ -125,12 +125,12 @@ object LiteralGenerator {
yield Literal.create(new Date(day * MILLIS_PER_DAY), DateType)
}
- lazy val timeLiteralGen: Gen[Literal] = {
- // Valid range for TimeType is [00:00:00, 23:59:59.999999]
- val minTime = nanosToMicros(localTimeToNanos(LocalTime.MIN)) *
NANOS_PER_MICROS
- val maxTime = nanosToMicros(localTimeToNanos(LocalTime.MAX)) *
NANOS_PER_MICROS
+ def timeLiteralGen(precision: Int): Gen[Literal] = {
+ // Valid range for TimeType is [00:00:00, 23:59:59.999999999] (nanoseconds
since midnight).
+ val minTime = localTimeToNanos(LocalTime.MIN)
+ val maxTime = localTimeToNanos(LocalTime.MAX)
for { t <- Gen.choose(minTime, maxTime) }
- yield Literal(t, TimeType())
+ yield Literal(DateTimeUtils.truncateTimeToPrecision(t, precision),
TimeType(precision))
}
private def millisGen = {
@@ -262,7 +262,7 @@ object LiteralGenerator {
case DoubleType => doubleLiteralGen
case FloatType => floatLiteralGen
case DateType => dateLiteralGen
- case _: TimeType => timeLiteralGen
+ case t: TimeType => timeLiteralGen(t.precision)
case TimestampType => timestampLiteralGen
case TimestampNTZType => timestampNTZLiteralGen
case t: TimestampNTZNanosType => timestampNTZNanosLiteralGen(t.precision)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/TimeExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/TimeExpressionsSuite.scala
index a806e2c9419c..6db6115a1e5e 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/TimeExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/TimeExpressionsSuite.scala
@@ -82,7 +82,7 @@ class TimeExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
assert(builtExprForTime.checkInputDataTypes().isSuccess)
// test TIME-typed child should build HoursOfTime for all allowed custom
precision values
- (TimeType.MIN_PRECISION to TimeType.MICROS_PRECISION).foreach { precision
=>
+ (TimeType.MIN_PRECISION to TimeType.MAX_PRECISION).foreach { precision =>
val timeExpr = Literal(localTime(12, 58, 59), TimeType(precision))
val builtExpr = HourExpressionBuilder.build("hour", Seq(timeExpr))
@@ -151,7 +151,7 @@ class TimeExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
assert(builtExprForTime.checkInputDataTypes().isSuccess)
// test TIME-typed child should build MinutesOfTime for all allowed custom
precision values
- (TimeType.MIN_PRECISION to TimeType.MICROS_PRECISION).foreach { precision
=>
+ (TimeType.MIN_PRECISION to TimeType.MAX_PRECISION).foreach { precision =>
val timeExpr = Literal(localTime(12, 58, 59), TimeType(precision))
val builtExpr = MinuteExpressionBuilder.build("minute", Seq(timeExpr))
@@ -315,6 +315,14 @@ class TimeExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
assert(expr.dataType == TimeType(2))
assert(expr.checkInputDataTypes() == TypeCheckSuccess)
+ // test nanosecond precisions 7, 8, 9 are valid
+ (TimeType.MICROS_PRECISION + 1 to TimeType.MAX_PRECISION).foreach { p =>
+ expr = CurrentTime(Literal(p))
+ assert(expr.precision == p, s"Precision should be $p")
+ assert(expr.dataType == TimeType(p))
+ assert(expr.checkInputDataTypes() == TypeCheckSuccess)
+ }
+
// test out of range precision => checkInputDataTypes fails
expr = CurrentTime(Literal(2 + 8))
assert(expr.checkInputDataTypes() ==
@@ -322,7 +330,7 @@ class TimeExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
errorSubClass = "VALUE_OUT_OF_RANGE",
messageParameters = Map(
"exprName" -> toSQLId("precision"),
- "valueRange" -> s"[${TimeType.MIN_PRECISION},
${TimeType.MICROS_PRECISION}]",
+ "valueRange" -> s"[${TimeType.MIN_PRECISION},
${TimeType.MAX_PRECISION}]",
"currentValue" -> toSQLValue(10, IntegerType)
)
)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/XmlExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/XmlExpressionsSuite.scala
index fb64b6fd7410..54bef646739f 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/XmlExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/XmlExpressionsSuite.scala
@@ -449,7 +449,10 @@ class XmlExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
(3, "14:30:45.123"),
(4, "14:30:45.1234"),
(5, "14:30:45.12345"),
- (6, "14:30:45.123456")
+ (6, "14:30:45.123456"),
+ (7, "14:30:45.1234567"),
+ (8, "14:30:45.12345678"),
+ (9, "23:59:59.999999999")
)
testData.foreach { case (precision, timeStr) =>
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
index 084bdbc460b7..6f9488ab4e52 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
@@ -64,6 +64,10 @@ class DataTypeParserSuite extends SparkFunSuite with
SQLHelper {
checkDataType("time(0) without time zone", TimeType(0))
checkDataType("TIME(6)", TimeType(6))
checkDataType("TIME(6) WITHOUT TIME ZONE", TimeType(6))
+ checkDataType("time(7)", TimeType(7))
+ checkDataType("TIME(8)", TimeType(8))
+ checkDataType("time(9)", TimeType(9))
+ checkDataType("TIME(9) WITHOUT TIME ZONE", TimeType(9))
checkDataType("timestamp", TimestampType)
checkDataType("TIMESTAMP WITH LOCAL TIME ZONE", TimestampType)
checkDataType("TIMESTAMP WITHOUT TIME ZONE", TimestampNTZType)
@@ -280,16 +284,16 @@ class DataTypeParserSuite extends SparkFunSuite with
SQLHelper {
test("unsupported precision of the time data type") {
checkError(
exception = intercept[SparkException] {
- CatalystSqlParser.parseDataType("time(9)")
+ CatalystSqlParser.parseDataType("time(10)")
},
condition = "UNSUPPORTED_TIME_PRECISION",
- parameters = Map("precision" -> "9"))
+ parameters = Map("precision" -> "10"))
checkError(
exception = intercept[SparkException] {
- CatalystSqlParser.parseDataType("time(8) without time zone")
+ CatalystSqlParser.parseDataType("time(11) without time zone")
},
condition = "UNSUPPORTED_TIME_PRECISION",
- parameters = Map("precision" -> "8"))
+ parameters = Map("precision" -> "11"))
checkError(
exception = intercept[ParseException] {
CatalystSqlParser.parseDataType("time(-1)")
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
index 636d16c78615..30bed5ca7649 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
@@ -1367,6 +1367,18 @@ class ExpressionParserSuite extends AnalysisTest {
assertEqual("tIme '12:13:14'", Literal(LocalTime.parse("12:13:14")))
assertEqual("TIME'23:59:59.999999'",
Literal(LocalTime.parse("23:59:59.999999")))
+ // ANSI SQL: the literal precision is the number of fractional-second
digits. 7-9 digits
+ // produce a nanosecond-precision TIME literal; <= 6 digits keep the
default precision (6).
+ assertEqual(
+ "TIME '12:34:56.1234567'",
+ Literal.create(LocalTime.parse("12:34:56.123456700"), TimeType(7)))
+ assertEqual(
+ "TIME '12:34:56.12345678'",
+ Literal.create(LocalTime.parse("12:34:56.123456780"), TimeType(8)))
+ assertEqual(
+ "TIME '23:59:59.999999999'",
+ Literal.create(LocalTime.parse("23:59:59.999999999"), TimeType(9)))
+
checkError(
exception = parseException("time '12-13.14'"),
condition = "INVALID_TYPED_LITERAL",
@@ -1376,6 +1388,17 @@ class ExpressionParserSuite extends AnalysisTest {
fragment = "time '12-13.14'",
start = 0,
stop = 14))
+
+ // More than 9 fractional-second digits is rejected.
+ checkError(
+ exception = parseException("TIME '12:34:56.1234567890'"),
+ condition = "INVALID_TIME_LITERAL_PRECISION",
+ sqlState = "22023",
+ parameters = Map("value" -> "'12:34:56.1234567890'"),
+ context = ExpectedContext(
+ fragment = "TIME '12:34:56.1234567890'",
+ start = 0,
+ stop = 25))
}
test("collate expression origin") {
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeTestUtils.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeTestUtils.scala
index b17a22778801..a3f9fcaf8265 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeTestUtils.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeTestUtils.scala
@@ -113,14 +113,17 @@ object DateTimeTestUtils {
result
}
- // Returns nanoseconds since midnight
+ // Returns nanoseconds since midnight. `micros` sets the microsecond part;
`nanos` adds the
+ // sub-microsecond remainder (digits 7-9), e.g. localTime(1, 2, 3, 987654,
321) is 01:02:03
+ // .987654321.
def localTime(
hour: Byte = 0,
minute: Byte = 0,
sec: Byte = 0,
- micros: Int = 0): Long = {
- val nanos = TimeUnit.MICROSECONDS.toNanos(micros).toInt
- val localTime = LocalTime.of(hour, minute, sec, nanos)
+ micros: Int = 0,
+ nanos: Int = 0): Long = {
+ val nanoOfSecond = TimeUnit.MICROSECONDS.toNanos(micros).toInt + nanos
+ val localTime = LocalTime.of(hour, minute, sec, nanoOfSecond)
localTimeToNanos(localTime)
}
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
index 8db143507819..0a6c9123a20e 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
@@ -1382,6 +1382,17 @@ class DateTimeUtilsSuite extends SparkFunSuite with
Matchers with SQLHelper {
Some(localTime(hour = 23, minute = 59, sec = 59, micros = 1)))
checkStringToTime("23:59:59.999999",
Some(localTime(hour = 23, minute = 59, sec = 59, micros = 999999)))
+ // Nanosecond resolution (7-9 fractional digits).
+ checkStringToTime("23:59:59.0000001",
+ Some(localTime(hour = 23, minute = 59, sec = 59, micros = 0, nanos =
100)))
+ checkStringToTime("23:59:59.00000012",
+ Some(localTime(hour = 23, minute = 59, sec = 59, micros = 0, nanos =
120)))
+ checkStringToTime("23:59:59.000000123",
+ Some(localTime(hour = 23, minute = 59, sec = 59, micros = 0, nanos =
123)))
+ checkStringToTime("23:59:59.999999999",
+ Some(localTime(hour = 23, minute = 59, sec = 59, micros = 999999, nanos
= 999)))
+ checkStringToTime("12:34:56.123456789",
+ Some(localTime(hour = 12, minute = 34, sec = 56, micros = 123456, nanos
= 789)))
checkStringToTime("1:2:3.0", Some(localTime(hour = 1, minute = 2, sec =
3)))
checkStringToTime("T1:02:3.04", Some(localTime(hour = 1, minute = 2, sec =
3, micros = 40000)))
@@ -1601,6 +1612,15 @@ class DateTimeUtilsSuite extends SparkFunSuite with
Matchers with SQLHelper {
localTime(23, 59, 59, 120000))
assert(truncateTimeToPrecision(localTime(23, 59, 59, 987654), 1) ==
localTime(23, 59, 59, 900000))
+ // Nanosecond precisions 7, 8, 9.
+ assert(truncateTimeToPrecision(localTime(23, 59, 59, 999999, 999), 9) ==
+ localTime(23, 59, 59, 999999, 999))
+ assert(truncateTimeToPrecision(localTime(23, 59, 59, 999999, 999), 8) ==
+ localTime(23, 59, 59, 999999, 990))
+ assert(truncateTimeToPrecision(localTime(23, 59, 59, 999999, 999), 7) ==
+ localTime(23, 59, 59, 999999, 900))
+ assert(truncateTimeToPrecision(localTime(23, 59, 59, 999999, 999), 6) ==
+ localTime(23, 59, 59, 999999))
}
test("add day-time interval to time") {
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimeFormatterSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimeFormatterSuite.scala
index 9a707d80d248..e7e7136202e9 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimeFormatterSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimeFormatterSuite.scala
@@ -41,7 +41,16 @@ class TimeFormatterSuite extends SparkFunSuite with
SQLHelper {
((12 * 3600 + 34 * 60 + 56) * 1000000000L + 789012000),
("23:59:59.000000", "HH:mm:ss.SSSSSS") -> (23 * 3600 + 59 * 60 + 59) *
1000000000L,
("23:59:59.999999", "HH:mm:ss.SSSSSS") ->
- ((23 * 3600 + 59 * 60 + 59) * 1000000000L + 999999000)
+ ((23 * 3600 + 59 * 60 + 59) * 1000000000L + 999999000),
+ // Nanosecond resolution (7-9 fractional digits).
+ ("12:34:56.7890123", "HH:mm:ss.SSSSSSS") ->
+ ((12 * 3600 + 34 * 60 + 56) * 1000000000L + 789012300),
+ ("12:34:56.78901234", "HH:mm:ss.SSSSSSSS") ->
+ ((12 * 3600 + 34 * 60 + 56) * 1000000000L + 789012340),
+ ("12:34:56.789012345", "HH:mm:ss.SSSSSSSSS") ->
+ ((12 * 3600 + 34 * 60 + 56) * 1000000000L + 789012345),
+ ("23:59:59.999999999", "HH:mm:ss.SSSSSSSSS") ->
+ ((23 * 3600 + 59 * 60 + 59) * 1000000000L + 999999999)
).foreach { case ((inputStr, pattern), expectedMicros) =>
val formatter = TimeFormatter(format = pattern, isParsing = true)
assert(formatter.parse(inputStr) === expectedMicros)
@@ -73,7 +82,12 @@ class TimeFormatterSuite extends SparkFunSuite with
SQLHelper {
"12:34:56.789012",
((23 * 3600 + 59 * 60 + 59) * 1000000000L, "HH:mm:ss.SSSSSS") ->
"23:59:59.000000",
((23 * 3600 + 59 * 60 + 59) * 1000000000L + 999999000,
"HH:mm:ss.SSSSSS") ->
- "23:59:59.999999"
+ "23:59:59.999999",
+ // Nanosecond resolution (7-9 fractional digits).
+ ((12 * 3600 + 34 * 60 + 56) * 1000000000L + 789012345,
"HH:mm:ss.SSSSSSSSS") ->
+ "12:34:56.789012345",
+ ((23 * 3600 + 59 * 60 + 59) * 1000000000L + 999999999,
"HH:mm:ss.SSSSSSSSS") ->
+ "23:59:59.999999999"
).foreach { case ((micros, pattern), expectedStr) =>
val formatter = TimeFormatter(format = pattern)
assert(formatter.format(micros) === expectedStr)
@@ -128,6 +142,15 @@ class TimeFormatterSuite extends SparkFunSuite with
SQLHelper {
assert(formatter.format(nanos) === tsStr)
assert(formatter.format(nanosToLocalTime(nanos)) === tsStr)
}
+ // Sub-microsecond fractions (digits 7-9) are formatted without trailing
zeros.
+ Seq(
+ 100L -> "00:00:00.0000001",
+ 120L -> "00:00:00.00000012",
+ 123L -> "00:00:00.000000123",
+ 999999999L -> "00:00:00.999999999").foreach { case (nanos, tsStr) =>
+ assert(formatter.format(nanos) === tsStr)
+ assert(formatter.format(nanosToLocalTime(nanos)) === tsStr)
+ }
}
test("missing am/pm field") {
@@ -163,7 +186,12 @@ class TimeFormatterSuite extends SparkFunSuite with
SQLHelper {
"00:00:00.000001" -> localTime(micros = 1),
"01:02:03" -> localTime(1, 2, 3),
"1:2:3.999999" -> localTime(1, 2, 3, 999999),
- "23:59:59.1" -> localTime(23, 59, 59, 100000)
+ "23:59:59.1" -> localTime(23, 59, 59, 100000),
+ // Nanosecond resolution (7-9 fractional digits).
+ "1:2:3.9999999" -> localTime(1, 2, 3, 999999, 900),
+ "1:2:3.99999999" -> localTime(1, 2, 3, 999999, 990),
+ "1:2:3.999999999" -> localTime(1, 2, 3, 999999, 999),
+ "23:59:59.123456789" -> localTime(23, 59, 59, 123456, 789)
).foreach { case (inputStr, micros) =>
assert(formatter.parse(inputStr) === micros)
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
index ff395ea70566..8b90f618dc6d 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
@@ -1459,7 +1459,7 @@ class DataTypeSuite extends SparkFunSuite with SQLHelper {
}
test("Parse time(n) as TimeType(n)") {
- 0 to 6 foreach { n =>
+ TimeType.MIN_PRECISION to TimeType.MAX_PRECISION foreach { n =>
assert(DataType.fromJson(s"\"time($n)\"") == TimeType(n))
val expectedStructType = StructType(Seq(StructField("t", TimeType(n))))
assert(DataType.fromDDL(s"t time($n)") == expectedStructType)
@@ -1467,10 +1467,10 @@ class DataTypeSuite extends SparkFunSuite with
SQLHelper {
checkError(
exception = intercept[SparkIllegalArgumentException] {
- DataType.fromJson("\"time(9)\"")
+ DataType.fromJson("\"time(10)\"")
},
condition = "INVALID_JSON_DATA_TYPE",
- parameters = Map("invalidType" -> "time(9)"))
+ parameters = Map("invalidType" -> "time(10)"))
checkError(
exception = intercept[ParseException] {
DataType.fromDDL("t time(-1)")
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java
index 8a72d2e5e731..35936dc80e88 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java
@@ -165,8 +165,17 @@ public class ParquetVectorUpdaterFactory {
return new LongUpdater();
} else if (canReadAsDecimal(descriptor, sparkType)) {
return new LongToDecimalUpdater(descriptor, (DecimalType) sparkType);
- } else if (sparkType instanceof TimeType) {
- return new LongAsNanosUpdater();
+ } else if (sparkType instanceof TimeType &&
+ isTimeTypeMatched(LogicalTypeAnnotation.TimeUnit.NANOS)) {
+ // TIME(NANOS) is stored as nanoseconds since midnight, matching the
internal
+ // representation, so no unit conversion is needed; the decoded
value is truncated to
+ // the requested precision (consistent with the row-based
ParquetRowConverter path).
+ return new TimeUpdater(((TimeType) sparkType).precision(), /*
fileStoresNanos = */ true);
+ } else if (sparkType instanceof TimeType &&
+ isTimeTypeMatched(LogicalTypeAnnotation.TimeUnit.MICROS)) {
+ // TIME(MICROS) is converted to nanoseconds, then truncated to the
requested precision
+ // (consistent with the row-based ParquetRowConverter path).
+ return new TimeUpdater(((TimeType) sparkType).precision(), /*
fileStoresNanos = */ false);
}
}
case FLOAT -> {
@@ -883,7 +892,40 @@ public class ParquetVectorUpdaterFactory {
}
}
- private static class LongAsNanosUpdater implements ParquetVectorUpdater {
+ // Reads an INT64 TIME column into the internal nanoseconds-since-midnight
representation and
+ // truncates it to the requested TimeType precision. `fileStoresNanos`
selects the on-disk unit:
+ // TIME(NANOS) stores nanos directly (identity), TIME(MICROS) stores micros
(converted to nanos).
+ // Mirrors the row-based ParquetRowConverter path so the vectorized and
non-vectorized readers
+ // agree on the decoded value, including when the requested precision is
lower than the on-disk
+ // value's precision.
+ // 10^k for k in [0, 9] (TimeType.NANOS_PRECISION), indexed by the
truncation scale
+ // (NANOS_PRECISION - p), used to truncate a nanosecond TIME value to the
requested
+ // fractional-second precision. Length - 1 equals TimeType.NANOS_PRECISION.
+ private static final long[] TIME_TRUNCATION_FACTORS = {
+ 1L, 10L, 100L, 1_000L, 10_000L, 100_000L,
+ 1_000_000L, 10_000_000L, 100_000_000L, 1_000_000_000L
+ };
+
+ private static class TimeUpdater implements ParquetVectorUpdater {
+ // The truncation step for the requested precision. The precision is
constant per column, so the
+ // factor is looked up once here rather than recomputed per value via the
math.pow in
+ // DateTimeUtils.truncateTimeToPrecision (this is the vectorized hot loop).
+ private final long truncationFactor;
+ private final boolean fileStoresNanos;
+
+ TimeUpdater(int precision, boolean fileStoresNanos) {
+ this.fileStoresNanos = fileStoresNanos;
+ // scale = NANOS_PRECISION - precision; NANOS_PRECISION ==
factors.length - 1.
+ int scale = TIME_TRUNCATION_FACTORS.length - 1 - precision;
+ this.truncationFactor = TIME_TRUNCATION_FACTORS[scale];
+ }
+
+ private long toTruncatedNanos(long value) {
+ long nanos = fileStoresNanos ? value :
DateTimeUtils.microsToNanos(value);
+ // Equivalent to DateTimeUtils.truncateTimeToPrecision with the factor
hoisted.
+ return (nanos / truncationFactor) * truncationFactor;
+ }
+
@Override
public void readValues(
int total,
@@ -892,7 +934,7 @@ public class ParquetVectorUpdaterFactory {
VectorizedValuesReader valuesReader) {
valuesReader.readLongs(total, values, offset);
for (int i = 0; i < total; i++) {
- values.putLong(offset + i,
DateTimeUtils.microsToNanos(values.getLong(offset + i)));
+ values.putLong(offset + i, toTruncatedNanos(values.getLong(offset +
i)));
}
}
@@ -906,7 +948,7 @@ public class ParquetVectorUpdaterFactory {
int offset,
WritableColumnVector values,
VectorizedValuesReader valuesReader) {
- values.putLong(offset,
DateTimeUtils.microsToNanos(valuesReader.readLong()));
+ values.putLong(offset, toTruncatedNanos(valuesReader.readLong()));
}
@Override
@@ -915,8 +957,8 @@ public class ParquetVectorUpdaterFactory {
WritableColumnVector values,
WritableColumnVector dictionaryIds,
Dictionary dictionary) {
- long micros = dictionary.decodeToLong(dictionaryIds.getDictId(offset));
- values.putLong(offset, DateTimeUtils.microsToNanos(micros));
+ long value = dictionary.decodeToLong(dictionaryIds.getDictId(offset));
+ values.putLong(offset, toTruncatedNanos(value));
}
}
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
index 971edfec3b11..eb7f1bd4d27d 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
@@ -166,9 +166,13 @@ public class VectorizedColumnReader {
}
case INT64: {
boolean isDecimal = sparkType instanceof DecimalType;
+ // TIME columns (both MICROS and NANOS) need per-value processing in
the updater: a unit
+ // conversion for MICROS and/or truncation to the requested precision.
Lazy dictionary
+ // decoding would bypass the updater, so it must be disabled for them.
boolean needsUpcast = (isDecimal &&
!DecimalType.is64BitDecimalType(sparkType)) ||
updaterFactory.isTimestampTypeMatched(TimeUnit.MILLIS) ||
- updaterFactory.isTimeTypeMatched(TimeUnit.MICROS);
+ updaterFactory.isTimeTypeMatched(TimeUnit.MICROS) ||
+ updaterFactory.isTimeTypeMatched(TimeUnit.NANOS);
boolean needsRebase =
updaterFactory.isTimestampTypeMatched(TimeUnit.MICROS) &&
!"CORRECTED".equals(datetimeRebaseMode);
isSupported = !needsUpcast && !needsRebase &&
!needsDecimalScaleRebase(sparkType);
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
index c3016d929ac9..2200179f5a9e 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
@@ -526,14 +526,19 @@ private[parquet] class ParquetRowConverter(
}
}
- case _: TimeType
- if
parquetType.getLogicalTypeAnnotation.isInstanceOf[TimeLogicalTypeAnnotation] &&
- parquetType.getLogicalTypeAnnotation
- .asInstanceOf[TimeLogicalTypeAnnotation].getUnit ==
TimeUnit.MICROS =>
+ case t: TimeType
+ if
parquetType.getLogicalTypeAnnotation.isInstanceOf[TimeLogicalTypeAnnotation] &&
{
+ val unit = parquetType.getLogicalTypeAnnotation
+ .asInstanceOf[TimeLogicalTypeAnnotation].getUnit
+ unit == TimeUnit.MICROS || unit == TimeUnit.NANOS
+ } =>
+ val fileStoresNanos = parquetType.getLogicalTypeAnnotation
+ .asInstanceOf[TimeLogicalTypeAnnotation].getUnit == TimeUnit.NANOS
+ val precision = t.precision
new ParquetPrimitiveConverter(updater) {
override def addLong(value: Long): Unit = {
- val nanos = DateTimeUtils.microsToNanos(value)
- this.updater.setLong(nanos)
+ val nanos = if (fileStoresNanos) value else
DateTimeUtils.microsToNanos(value)
+ this.updater.setLong(DateTimeUtils.truncateTimeToPrecision(nanos,
precision))
}
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala
index ba5261200464..a0acc40f9e01 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala
@@ -342,6 +342,9 @@ class ParquetToSparkSchemaConverter(
case time: TimeLogicalTypeAnnotation
if time.getUnit == TimeUnit.MICROS && !time.isAdjustedToUTC =>
TimeType(TimeType.MICROS_PRECISION)
+ case time: TimeLogicalTypeAnnotation
+ if time.getUnit == TimeUnit.NANOS && !time.isAdjustedToUTC =>
+ TimeType(TimeType.NANOS_PRECISION)
case _ => illegalType()
}
@@ -710,9 +713,11 @@ class SparkToParquetSchemaConverter(
Types.primitive(INT32, repetition)
.as(LogicalTypeAnnotation.dateType()).named(field.name)
- case _: TimeType =>
+ case t: TimeType =>
+ // Precision 0..6 is stored as TIME(MICROS); precision 7..9 as
TIME(NANOS).
+ val unit = if (t.precision > TimeType.MICROS_PRECISION) TimeUnit.NANOS
else TimeUnit.MICROS
Types.primitive(INT64, repetition)
- .as(LogicalTypeAnnotation.timeType(false,
TimeUnit.MICROS)).named(field.name)
+ .as(LogicalTypeAnnotation.timeType(false, unit)).named(field.name)
// NOTE: Spark SQL can write timestamp values to Parquet using INT96,
TIMESTAMP_MICROS or
// TIMESTAMP_MILLIS. TIMESTAMP_MICROS is recommended but INT96 is the
default to keep the
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala
index 18c0a47facda..c7d7426a94ea 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala
@@ -307,6 +307,11 @@ class ParquetWriteSupport extends
WriteSupport[InternalRow] with Logging {
recordConsumer.addLong(
timestampNanosToEpochNanos(row.getTimestampNTZNanos(ordinal),
isNtz = true))
+ case t: TimeType if t.precision > TimeType.MICROS_PRECISION =>
+ // Precision 7..9 is stored as TIME(NANOS); internal storage is
already nanos.
+ (row: SpecializedGetters, ordinal: Int) =>
+ recordConsumer.addLong(row.getLong(ordinal))
+
case _: TimeType =>
(row: SpecializedGetters, ordinal: Int) =>
recordConsumer.addLong(DateTimeUtils.nanosToMicros(row.getLong(ordinal)))
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimeTypeParquetOps.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimeTypeParquetOps.scala
index 96c7bc30a1da..f155d35378a8 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimeTypeParquetOps.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimeTypeParquetOps.scala
@@ -33,27 +33,35 @@ import org.apache.spark.sql.types.{DataType, TimeType}
/**
* Parquet operations for TimeType.
*
- * TimeType is a primitive Long-backed type stored in Parquet as INT64 with the
- * TIME(isAdjustedToUTC=false, unit=MICROS) logical type annotation.
+ * TimeType is a primitive Long-backed type stored in Parquet as INT64 with a
+ * TIME(isAdjustedToUTC=false) logical type annotation. The annotation unit
depends on the
+ * requested precision: precision 0..6 uses TIME(MICROS), precision 7..9 uses
TIME(NANOS).
*
* IMPORTANT - internal vs Parquet representation:
* - Spark internal: nanoseconds since midnight (Long)
- * - Parquet storage: microseconds since midnight (INT64)
- * - Write path: nanos -> micros (DateTimeUtils.nanosToMicros)
- * - Read path: micros -> nanos (DateTimeUtils.microsToNanos)
+ * - Parquet storage: microseconds (precision <= 6) or nanoseconds
(precision >= 7) since midnight
+ * - Write path: nanos -> micros (DateTimeUtils.nanosToMicros) for
TIME(MICROS); identity for
+ * TIME(NANOS)
+ * - Read path: micros -> nanos (DateTimeUtils.microsToNanos) for
TIME(MICROS); identity for
+ * TIME(NANOS), then truncated to the requested precision
*
* @param t the TimeType with precision information
* @since 4.3.0
*/
case class TimeTypeParquetOps(t: TimeType) extends ParquetTypeOps {
+ // Precision 7..9 is stored as TIME(NANOS); precision 0..6 as TIME(MICROS).
+ private def storesNanos: Boolean = t.precision > TimeType.MICROS_PRECISION
+
// ==================== Schema Conversion ====================
override def convertToParquetType(
- fieldName: String, repetition: Repetition, inShredded: Boolean): Type =
+ fieldName: String, repetition: Repetition, inShredded: Boolean): Type = {
+ val unit = if (storesNanos) TimeUnit.NANOS else TimeUnit.MICROS
Types.primitive(INT64, repetition)
- .as(LogicalTypeAnnotation.timeType(false, TimeUnit.MICROS))
+ .as(LogicalTypeAnnotation.timeType(false, unit))
.named(fieldName)
+ }
// ==================== Value Write ====================
@@ -63,8 +71,14 @@ case class TimeTypeParquetOps(t: TimeType) extends
ParquetTypeOps {
): (SpecializedGetters, Int) => Unit =
// Evaluate the supplier at write time (not creation time) because
recordConsumer
// is null during init() and set later in prepareForWrite().
- (row: SpecializedGetters, ordinal: Int) =>
-
recordConsumer().addLong(DateTimeUtils.nanosToMicros(row.getLong(ordinal)))
+ if (storesNanos) {
+ // Internal storage is already nanoseconds since midnight, so write it
unchanged.
+ (row: SpecializedGetters, ordinal: Int) =>
+ recordConsumer().addLong(row.getLong(ordinal))
+ } else {
+ (row: SpecializedGetters, ordinal: Int) =>
+
recordConsumer().addLong(DateTimeUtils.nanosToMicros(row.getLong(ordinal)))
+ }
// ==================== Row-Based Read ====================
@@ -73,14 +87,16 @@ case class TimeTypeParquetOps(t: TimeType) extends
ParquetTypeOps {
updater: ParentContainerUpdater): Converter with
HasParentContainerUpdater = {
// Framework-first dispatch in ParquetRowConverter routes here whenever the
// requested Spark type is TimeType, regardless of the actual Parquet
encoding.
- // Without this guard, files whose column is raw INT64, INT64 TIME(NANOS),
- // INT64 TIMESTAMP(MICROS), INT32 TIME(MILLIS), etc. would silently decode
as
- // microsToNanos(value) and produce wrong results. Mirrors the inline guard
- // that existed in ParquetRowConverter before the framework dispatch.
+ // Without this guard, files whose column is raw INT64, INT64
TIMESTAMP(MICROS),
+ // INT32 TIME(MILLIS), etc. would silently decode as microsToNanos(value)
and
+ // produce wrong results.
TimeTypeParquetOps.requireCompatibleParquetType(t, parquetType)
+ val fileStoresNanos = TimeTypeParquetOps.isNanosTime(parquetType)
+ val precision = t.precision
new ParquetPrimitiveConverter(updater) {
override def addLong(value: Long): Unit = {
- this.updater.setLong(DateTimeUtils.microsToNanos(value))
+ val nanos = if (fileStoresNanos) value else
DateTimeUtils.microsToNanos(value)
+ this.updater.setLong(DateTimeUtils.truncateTimeToPrecision(nanos,
precision))
}
}
}
@@ -93,14 +109,27 @@ case class TimeTypeParquetOps(t: TimeType) extends
ParquetTypeOps {
private[ops] object TimeTypeParquetOps {
/**
- * Validates that a Parquet field can be decoded as TimeType. TimeType is
written
- * as INT64 with TIME(MICROS, isAdjustedToUTC=false). On read, any INT64
TIME(MICROS)
- * column is accepted regardless of the isAdjustedToUTC flag: Spark's
zone-less TimeType
- * decodes the raw micros-of-day identically either way, matching the legacy
- * ParquetRowConverter guard (see SPARK-57416). Any other encoding (raw
INT64, INT64
- * TIME(NANOS), INT32 TIME(MILLIS), INT64 TIMESTAMP(_), decimal-annotated,
etc.) cannot
- * be decoded as TimeType - throw the same error as the legacy
ParquetRowConverter path
- * so reads fail loudly instead of silently misinterpreting bytes.
+ * Whether the Parquet field is an INT64 TIME(NANOS) column. The
isAdjustedToUTC flag is
+ * intentionally ignored: Spark's TimeType is zone-less, so a TIME(NANOS)
value decodes to the
+ * same nanos-of-day regardless of the flag (consistent with
requireCompatibleParquetType and
+ * the legacy reader; see SPARK-57416).
+ */
+ private[ops] def isNanosTime(parquetType: Type): Boolean =
+ parquetType.getLogicalTypeAnnotation match {
+ case t: LogicalTypeAnnotation.TimeLogicalTypeAnnotation =>
+ t.getUnit == TimeUnit.NANOS
+ case _ => false
+ }
+
+ /**
+ * Validates that a Parquet field can be decoded as TimeType. TimeType is
written as INT64 with
+ * TIME(MICROS, isAdjustedToUTC=false) for precision 0..6 and TIME(NANOS,
isAdjustedToUTC=false)
+ * for precision 7..9. On read, any INT64 TIME(MICROS) or TIME(NANOS) column
is accepted
+ * regardless of the isAdjustedToUTC flag: Spark's zone-less TimeType
decodes the raw
+ * time-of-day identically either way, matching the legacy
ParquetRowConverter guard (see
+ * SPARK-57416). Any other encoding (raw INT64, INT32 TIME(MILLIS), INT64
TIMESTAMP(_),
+ * decimal-annotated, etc.) cannot be decoded as TimeType - throw the same
error as the legacy
+ * ParquetRowConverter path so reads fail loudly instead of silently
misinterpreting bytes.
*/
private[ops] def requireCompatibleParquetType(
sparkType: TimeType, parquetType: Type): Unit = {
@@ -108,13 +137,13 @@ private[ops] object TimeTypeParquetOps {
parquetType.asPrimitiveType.getPrimitiveTypeName == INT64 &&
(parquetType.getLogicalTypeAnnotation match {
case t: LogicalTypeAnnotation.TimeLogicalTypeAnnotation =>
- // Accept both isAdjustedToUTC=false and =true. Spark's TimeType is
zone-less
- // local time, so the UTC-adjustment flag carries no extra
information on read:
- // the raw micros-of-day value decodes identically either way.
Mirroring the
- // legacy ParquetRowConverter guard (which only checked the TIME
annotation and
- // the MICROS unit) keeps the framework row-based read path
consistent with both
- // the legacy row-based reader and the still-lenient vectorized
reader. SPARK-57416.
- t.getUnit == TimeUnit.MICROS
+ // Accept both MICROS (precision 0..6) and NANOS (precision 7..9),
and both
+ // isAdjustedToUTC=false and =true. Spark's TimeType is zone-less
local time, so the
+ // UTC-adjustment flag carries no extra information on read: the raw
time-of-day value
+ // decodes identically either way. Mirroring the legacy
ParquetRowConverter guard keeps
+ // the framework row-based read path consistent with the legacy and
vectorized readers.
+ // SPARK-57416.
+ t.getUnit == TimeUnit.MICROS || t.getUnit == TimeUnit.NANOS
case _ => false
})
if (!ok) {
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/cast.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/cast.sql.out
index a3c8e894f8a9..2ad6e040585b 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/cast.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/cast.sql.out
@@ -1676,14 +1676,14 @@ Project [cast(23:59:59.999999 as decimal(11,6)) AS
CAST(TIME '23:59:59.999999' A
-- !query
SELECT CAST(time '23:59:59.999999999' AS decimal(14, 9))
-- !query analysis
-Project [cast(23:59:59.999999 as decimal(14,9)) AS CAST(TIME '23:59:59.999999'
AS DECIMAL(14,9))#x]
+Project [cast(23:59:59.999999999 as decimal(14,9)) AS CAST(TIME
'23:59:59.999999999' AS DECIMAL(14,9))#x]
+- OneRowRelation
-- !query
SELECT CAST(time '23:59:59.999999999' AS decimal(20, 10))
-- !query analysis
-Project [cast(23:59:59.999999 as decimal(20,10)) AS CAST(TIME
'23:59:59.999999' AS DECIMAL(20,10))#x]
+Project [cast(23:59:59.999999999 as decimal(20,10)) AS CAST(TIME
'23:59:59.999999999' AS DECIMAL(20,10))#x]
+- OneRowRelation
@@ -1732,5 +1732,5 @@ Project [cast(23:59:59.999999 as decimal(11,5)) AS
CAST(TIME '23:59:59.999999' A
-- !query
SELECT CAST(time '23:59:59.999999999' AS decimal(14, 8))
-- !query analysis
-Project [cast(23:59:59.999999 as decimal(14,8)) AS CAST(TIME '23:59:59.999999'
AS DECIMAL(14,8))#x]
+Project [cast(23:59:59.999999999 as decimal(14,8)) AS CAST(TIME
'23:59:59.999999999' AS DECIMAL(14,8))#x]
+- OneRowRelation
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/nonansi/cast.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/nonansi/cast.sql.out
index b18f5739fd68..c48776a1de31 100644
---
a/sql/core/src/test/resources/sql-tests/analyzer-results/nonansi/cast.sql.out
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/nonansi/cast.sql.out
@@ -1523,14 +1523,14 @@ Project [cast(23:59:59.999999 as decimal(11,6)) AS
CAST(TIME '23:59:59.999999' A
-- !query
SELECT CAST(time '23:59:59.999999999' AS decimal(14, 9))
-- !query analysis
-Project [cast(23:59:59.999999 as decimal(14,9)) AS CAST(TIME '23:59:59.999999'
AS DECIMAL(14,9))#x]
+Project [cast(23:59:59.999999999 as decimal(14,9)) AS CAST(TIME
'23:59:59.999999999' AS DECIMAL(14,9))#x]
+- OneRowRelation
-- !query
SELECT CAST(time '23:59:59.999999999' AS decimal(20, 10))
-- !query analysis
-Project [cast(23:59:59.999999 as decimal(20,10)) AS CAST(TIME
'23:59:59.999999' AS DECIMAL(20,10))#x]
+Project [cast(23:59:59.999999999 as decimal(20,10)) AS CAST(TIME
'23:59:59.999999999' AS DECIMAL(20,10))#x]
+- OneRowRelation
@@ -1579,5 +1579,5 @@ Project [cast(23:59:59.999999 as decimal(11,5)) AS
CAST(TIME '23:59:59.999999' A
-- !query
SELECT CAST(time '23:59:59.999999999' AS decimal(14, 8))
-- !query analysis
-Project [cast(23:59:59.999999 as decimal(14,8)) AS CAST(TIME '23:59:59.999999'
AS DECIMAL(14,8))#x]
+Project [cast(23:59:59.999999999 as decimal(14,8)) AS CAST(TIME
'23:59:59.999999999' AS DECIMAL(14,8))#x]
+- OneRowRelation
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/time.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/time.sql.out
index 8c5c55bfa0a0..d69a6df9eb5e 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/time.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/time.sql.out
@@ -959,35 +959,35 @@ Project [time_trunc(MICROSECOND, 12:34:56.123456) AS
time_trunc(MICROSECOND, TIM
-- !query
SELECT time_trunc('HOUR', time'12:34:56.123456789')
-- !query analysis
-Project [time_trunc(HOUR, 12:34:56.123456) AS time_trunc(HOUR, TIME
'12:34:56.123456')#x]
+Project [time_trunc(HOUR, 12:34:56.123456789) AS time_trunc(HOUR, TIME
'12:34:56.123456789')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('MINUTE', time'12:34:56.123456789')
-- !query analysis
-Project [time_trunc(MINUTE, 12:34:56.123456) AS time_trunc(MINUTE, TIME
'12:34:56.123456')#x]
+Project [time_trunc(MINUTE, 12:34:56.123456789) AS time_trunc(MINUTE, TIME
'12:34:56.123456789')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('SECOND', time'12:34:56.123456789')
-- !query analysis
-Project [time_trunc(SECOND, 12:34:56.123456) AS time_trunc(SECOND, TIME
'12:34:56.123456')#x]
+Project [time_trunc(SECOND, 12:34:56.123456789) AS time_trunc(SECOND, TIME
'12:34:56.123456789')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('MILLISECOND', time'12:34:56.123456789')
-- !query analysis
-Project [time_trunc(MILLISECOND, 12:34:56.123456) AS time_trunc(MILLISECOND,
TIME '12:34:56.123456')#x]
+Project [time_trunc(MILLISECOND, 12:34:56.123456789) AS
time_trunc(MILLISECOND, TIME '12:34:56.123456789')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('MICROSECOND', time'12:34:56.123456789')
-- !query analysis
-Project [time_trunc(MICROSECOND, 12:34:56.123456) AS time_trunc(MICROSECOND,
TIME '12:34:56.123456')#x]
+Project [time_trunc(MICROSECOND, 12:34:56.123456789) AS
time_trunc(MICROSECOND, TIME '12:34:56.123456789')#x]
+- OneRowRelation
@@ -1064,70 +1064,70 @@ Project [time_trunc(MICROSECOND, 00:00:00) AS
time_trunc(MICROSECOND, TIME '00:0
-- !query
SELECT time_trunc('HOUR', time'00:00:00.000000001')
-- !query analysis
-Project [time_trunc(HOUR, 00:00:00) AS time_trunc(HOUR, TIME '00:00:00')#x]
+Project [time_trunc(HOUR, 00:00:00.000000001) AS time_trunc(HOUR, TIME
'00:00:00.000000001')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('MINUTE', time'00:00:00.000000001')
-- !query analysis
-Project [time_trunc(MINUTE, 00:00:00) AS time_trunc(MINUTE, TIME '00:00:00')#x]
+Project [time_trunc(MINUTE, 00:00:00.000000001) AS time_trunc(MINUTE, TIME
'00:00:00.000000001')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('SECOND', time'00:00:00.000000001')
-- !query analysis
-Project [time_trunc(SECOND, 00:00:00) AS time_trunc(SECOND, TIME '00:00:00')#x]
+Project [time_trunc(SECOND, 00:00:00.000000001) AS time_trunc(SECOND, TIME
'00:00:00.000000001')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('MILLISECOND', time'00:00:00.000000001')
-- !query analysis
-Project [time_trunc(MILLISECOND, 00:00:00) AS time_trunc(MILLISECOND, TIME
'00:00:00')#x]
+Project [time_trunc(MILLISECOND, 00:00:00.000000001) AS
time_trunc(MILLISECOND, TIME '00:00:00.000000001')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('MICROSECOND', time'00:00:00.000000001')
-- !query analysis
-Project [time_trunc(MICROSECOND, 00:00:00) AS time_trunc(MICROSECOND, TIME
'00:00:00')#x]
+Project [time_trunc(MICROSECOND, 00:00:00.000000001) AS
time_trunc(MICROSECOND, TIME '00:00:00.000000001')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('HOUR', time'23:59:59.999999999')
-- !query analysis
-Project [time_trunc(HOUR, 23:59:59.999999) AS time_trunc(HOUR, TIME
'23:59:59.999999')#x]
+Project [time_trunc(HOUR, 23:59:59.999999999) AS time_trunc(HOUR, TIME
'23:59:59.999999999')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('MINUTE', time'23:59:59.999999999')
-- !query analysis
-Project [time_trunc(MINUTE, 23:59:59.999999) AS time_trunc(MINUTE, TIME
'23:59:59.999999')#x]
+Project [time_trunc(MINUTE, 23:59:59.999999999) AS time_trunc(MINUTE, TIME
'23:59:59.999999999')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('SECOND', time'23:59:59.999999999')
-- !query analysis
-Project [time_trunc(SECOND, 23:59:59.999999) AS time_trunc(SECOND, TIME
'23:59:59.999999')#x]
+Project [time_trunc(SECOND, 23:59:59.999999999) AS time_trunc(SECOND, TIME
'23:59:59.999999999')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('MILLISECOND', time'23:59:59.999999999')
-- !query analysis
-Project [time_trunc(MILLISECOND, 23:59:59.999999) AS time_trunc(MILLISECOND,
TIME '23:59:59.999999')#x]
+Project [time_trunc(MILLISECOND, 23:59:59.999999999) AS
time_trunc(MILLISECOND, TIME '23:59:59.999999999')#x]
+- OneRowRelation
-- !query
SELECT time_trunc('MICROSECOND', time'23:59:59.999999999')
-- !query analysis
-Project [time_trunc(MICROSECOND, 23:59:59.999999) AS time_trunc(MICROSECOND,
TIME '23:59:59.999999')#x]
+Project [time_trunc(MICROSECOND, 23:59:59.999999999) AS
time_trunc(MICROSECOND, TIME '23:59:59.999999999')#x]
+- OneRowRelation
@@ -1533,35 +1533,35 @@ Project [time_diff(MICROSECOND, 00:00:00,
12:34:56.123456) AS time_diff(MICROSEC
-- !query
SELECT time_diff('HOUR', time'00:00:00', time'12:34:56.123456789')
-- !query analysis
-Project [time_diff(HOUR, 00:00:00, 12:34:56.123456) AS time_diff(HOUR, TIME
'00:00:00', TIME '12:34:56.123456')#xL]
+Project [time_diff(HOUR, 00:00:00, 12:34:56.123456789) AS time_diff(HOUR, TIME
'00:00:00', TIME '12:34:56.123456789')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('MINUTE', time'00:00:00', time'12:34:56.123456789')
-- !query analysis
-Project [time_diff(MINUTE, 00:00:00, 12:34:56.123456) AS time_diff(MINUTE,
TIME '00:00:00', TIME '12:34:56.123456')#xL]
+Project [time_diff(MINUTE, 00:00:00, 12:34:56.123456789) AS time_diff(MINUTE,
TIME '00:00:00', TIME '12:34:56.123456789')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('SECOND', time'00:00:00', time'12:34:56.123456789')
-- !query analysis
-Project [time_diff(SECOND, 00:00:00, 12:34:56.123456) AS time_diff(SECOND,
TIME '00:00:00', TIME '12:34:56.123456')#xL]
+Project [time_diff(SECOND, 00:00:00, 12:34:56.123456789) AS time_diff(SECOND,
TIME '00:00:00', TIME '12:34:56.123456789')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('MILLISECOND', time'00:00:00', time'12:34:56.123456789')
-- !query analysis
-Project [time_diff(MILLISECOND, 00:00:00, 12:34:56.123456) AS
time_diff(MILLISECOND, TIME '00:00:00', TIME '12:34:56.123456')#xL]
+Project [time_diff(MILLISECOND, 00:00:00, 12:34:56.123456789) AS
time_diff(MILLISECOND, TIME '00:00:00', TIME '12:34:56.123456789')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('MICROSECOND', time'00:00:00', time'12:34:56.123456789')
-- !query analysis
-Project [time_diff(MICROSECOND, 00:00:00, 12:34:56.123456) AS
time_diff(MICROSECOND, TIME '00:00:00', TIME '12:34:56.123456')#xL]
+Project [time_diff(MICROSECOND, 00:00:00, 12:34:56.123456789) AS
time_diff(MICROSECOND, TIME '00:00:00', TIME '12:34:56.123456789')#xL]
+- OneRowRelation
@@ -1638,70 +1638,70 @@ Project [time_diff(MICROSECOND, 00:00:00, 00:00:00) AS
time_diff(MICROSECOND, TI
-- !query
SELECT time_diff('HOUR', time'00:00:00', time'00:00:00.000000001')
-- !query analysis
-Project [time_diff(HOUR, 00:00:00, 00:00:00) AS time_diff(HOUR, TIME
'00:00:00', TIME '00:00:00')#xL]
+Project [time_diff(HOUR, 00:00:00, 00:00:00.000000001) AS time_diff(HOUR, TIME
'00:00:00', TIME '00:00:00.000000001')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('MINUTE', time'00:00:00', time'00:00:00.000000001')
-- !query analysis
-Project [time_diff(MINUTE, 00:00:00, 00:00:00) AS time_diff(MINUTE, TIME
'00:00:00', TIME '00:00:00')#xL]
+Project [time_diff(MINUTE, 00:00:00, 00:00:00.000000001) AS time_diff(MINUTE,
TIME '00:00:00', TIME '00:00:00.000000001')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('SECOND', time'00:00:00', time'00:00:00.000000001')
-- !query analysis
-Project [time_diff(SECOND, 00:00:00, 00:00:00) AS time_diff(SECOND, TIME
'00:00:00', TIME '00:00:00')#xL]
+Project [time_diff(SECOND, 00:00:00, 00:00:00.000000001) AS time_diff(SECOND,
TIME '00:00:00', TIME '00:00:00.000000001')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('MILLISECOND', time'00:00:00', time'00:00:00.000000001')
-- !query analysis
-Project [time_diff(MILLISECOND, 00:00:00, 00:00:00) AS time_diff(MILLISECOND,
TIME '00:00:00', TIME '00:00:00')#xL]
+Project [time_diff(MILLISECOND, 00:00:00, 00:00:00.000000001) AS
time_diff(MILLISECOND, TIME '00:00:00', TIME '00:00:00.000000001')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('MICROSECOND', time'00:00:00', time'00:00:00.000000001')
-- !query analysis
-Project [time_diff(MICROSECOND, 00:00:00, 00:00:00) AS time_diff(MICROSECOND,
TIME '00:00:00', TIME '00:00:00')#xL]
+Project [time_diff(MICROSECOND, 00:00:00, 00:00:00.000000001) AS
time_diff(MICROSECOND, TIME '00:00:00', TIME '00:00:00.000000001')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('HOUR', time'00:00:00', time'23:59:59.999999999')
-- !query analysis
-Project [time_diff(HOUR, 00:00:00, 23:59:59.999999) AS time_diff(HOUR, TIME
'00:00:00', TIME '23:59:59.999999')#xL]
+Project [time_diff(HOUR, 00:00:00, 23:59:59.999999999) AS time_diff(HOUR, TIME
'00:00:00', TIME '23:59:59.999999999')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('MINUTE', time'00:00:00', time'23:59:59.999999999')
-- !query analysis
-Project [time_diff(MINUTE, 00:00:00, 23:59:59.999999) AS time_diff(MINUTE,
TIME '00:00:00', TIME '23:59:59.999999')#xL]
+Project [time_diff(MINUTE, 00:00:00, 23:59:59.999999999) AS time_diff(MINUTE,
TIME '00:00:00', TIME '23:59:59.999999999')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('SECOND', time'00:00:00', time'23:59:59.999999999')
-- !query analysis
-Project [time_diff(SECOND, 00:00:00, 23:59:59.999999) AS time_diff(SECOND,
TIME '00:00:00', TIME '23:59:59.999999')#xL]
+Project [time_diff(SECOND, 00:00:00, 23:59:59.999999999) AS time_diff(SECOND,
TIME '00:00:00', TIME '23:59:59.999999999')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('MILLISECOND', time'00:00:00', time'23:59:59.999999999')
-- !query analysis
-Project [time_diff(MILLISECOND, 00:00:00, 23:59:59.999999) AS
time_diff(MILLISECOND, TIME '00:00:00', TIME '23:59:59.999999')#xL]
+Project [time_diff(MILLISECOND, 00:00:00, 23:59:59.999999999) AS
time_diff(MILLISECOND, TIME '00:00:00', TIME '23:59:59.999999999')#xL]
+- OneRowRelation
-- !query
SELECT time_diff('MICROSECOND', time'00:00:00', time'23:59:59.999999999')
-- !query analysis
-Project [time_diff(MICROSECOND, 00:00:00, 23:59:59.999999) AS
time_diff(MICROSECOND, TIME '00:00:00', TIME '23:59:59.999999')#xL]
+Project [time_diff(MICROSECOND, 00:00:00, 23:59:59.999999999) AS
time_diff(MICROSECOND, TIME '00:00:00', TIME '23:59:59.999999999')#xL]
+- OneRowRelation
diff --git a/sql/core/src/test/resources/sql-tests/results/cast.sql.out
b/sql/core/src/test/resources/sql-tests/results/cast.sql.out
index 24f9668b3b17..a6d3cee116aa 100644
--- a/sql/core/src/test/resources/sql-tests/results/cast.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/cast.sql.out
@@ -2753,17 +2753,17 @@ struct<CAST(TIME '23:59:59.999999' AS
DECIMAL(11,6)):decimal(11,6)>
-- !query
SELECT CAST(time '23:59:59.999999999' AS decimal(14, 9))
-- !query schema
-struct<CAST(TIME '23:59:59.999999' AS DECIMAL(14,9)):decimal(14,9)>
+struct<CAST(TIME '23:59:59.999999999' AS DECIMAL(14,9)):decimal(14,9)>
-- !query output
-86399.999999000
+86399.999999999
-- !query
SELECT CAST(time '23:59:59.999999999' AS decimal(20, 10))
-- !query schema
-struct<CAST(TIME '23:59:59.999999' AS DECIMAL(20,10)):decimal(20,10)>
+struct<CAST(TIME '23:59:59.999999999' AS DECIMAL(20,10)):decimal(20,10)>
-- !query output
-86399.9999990000
+86399.9999999990
-- !query
@@ -2868,6 +2868,6 @@ struct<CAST(TIME '23:59:59.999999' AS
DECIMAL(11,5)):decimal(11,5)>
-- !query
SELECT CAST(time '23:59:59.999999999' AS decimal(14, 8))
-- !query schema
-struct<CAST(TIME '23:59:59.999999' AS DECIMAL(14,8)):decimal(14,8)>
+struct<CAST(TIME '23:59:59.999999999' AS DECIMAL(14,8)):decimal(14,8)>
-- !query output
-86399.99999900
+86400.00000000
diff --git a/sql/core/src/test/resources/sql-tests/results/nonansi/cast.sql.out
b/sql/core/src/test/resources/sql-tests/results/nonansi/cast.sql.out
index 4d66796a577b..1bc9c12b0421 100644
--- a/sql/core/src/test/resources/sql-tests/results/nonansi/cast.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/nonansi/cast.sql.out
@@ -1804,17 +1804,17 @@ struct<CAST(TIME '23:59:59.999999' AS
DECIMAL(11,6)):decimal(11,6)>
-- !query
SELECT CAST(time '23:59:59.999999999' AS decimal(14, 9))
-- !query schema
-struct<CAST(TIME '23:59:59.999999' AS DECIMAL(14,9)):decimal(14,9)>
+struct<CAST(TIME '23:59:59.999999999' AS DECIMAL(14,9)):decimal(14,9)>
-- !query output
-86399.999999000
+86399.999999999
-- !query
SELECT CAST(time '23:59:59.999999999' AS decimal(20, 10))
-- !query schema
-struct<CAST(TIME '23:59:59.999999' AS DECIMAL(20,10)):decimal(20,10)>
+struct<CAST(TIME '23:59:59.999999999' AS DECIMAL(20,10)):decimal(20,10)>
-- !query output
-86399.9999990000
+86399.9999999990
-- !query
@@ -1868,6 +1868,6 @@ struct<CAST(TIME '23:59:59.999999' AS
DECIMAL(11,5)):decimal(11,5)>
-- !query
SELECT CAST(time '23:59:59.999999999' AS decimal(14, 8))
-- !query schema
-struct<CAST(TIME '23:59:59.999999' AS DECIMAL(14,8)):decimal(14,8)>
+struct<CAST(TIME '23:59:59.999999999' AS DECIMAL(14,8)):decimal(14,8)>
-- !query output
-86399.99999900
+86400.00000000
diff --git a/sql/core/src/test/resources/sql-tests/results/time.sql.out
b/sql/core/src/test/resources/sql-tests/results/time.sql.out
index 5767be09ece5..c61c37f27cd3 100644
--- a/sql/core/src/test/resources/sql-tests/results/time.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/time.sql.out
@@ -1122,7 +1122,7 @@ struct<time_trunc(MICROSECOND, TIME
'12:34:56.123456'):time(6)>
-- !query
SELECT time_trunc('HOUR', time'12:34:56.123456789')
-- !query schema
-struct<time_trunc(HOUR, TIME '12:34:56.123456'):time(6)>
+struct<time_trunc(HOUR, TIME '12:34:56.123456789'):time(9)>
-- !query output
12:00:00
@@ -1130,7 +1130,7 @@ struct<time_trunc(HOUR, TIME '12:34:56.123456'):time(6)>
-- !query
SELECT time_trunc('MINUTE', time'12:34:56.123456789')
-- !query schema
-struct<time_trunc(MINUTE, TIME '12:34:56.123456'):time(6)>
+struct<time_trunc(MINUTE, TIME '12:34:56.123456789'):time(9)>
-- !query output
12:34:00
@@ -1138,7 +1138,7 @@ struct<time_trunc(MINUTE, TIME '12:34:56.123456'):time(6)>
-- !query
SELECT time_trunc('SECOND', time'12:34:56.123456789')
-- !query schema
-struct<time_trunc(SECOND, TIME '12:34:56.123456'):time(6)>
+struct<time_trunc(SECOND, TIME '12:34:56.123456789'):time(9)>
-- !query output
12:34:56
@@ -1146,7 +1146,7 @@ struct<time_trunc(SECOND, TIME '12:34:56.123456'):time(6)>
-- !query
SELECT time_trunc('MILLISECOND', time'12:34:56.123456789')
-- !query schema
-struct<time_trunc(MILLISECOND, TIME '12:34:56.123456'):time(6)>
+struct<time_trunc(MILLISECOND, TIME '12:34:56.123456789'):time(9)>
-- !query output
12:34:56.123
@@ -1154,7 +1154,7 @@ struct<time_trunc(MILLISECOND, TIME
'12:34:56.123456'):time(6)>
-- !query
SELECT time_trunc('MICROSECOND', time'12:34:56.123456789')
-- !query schema
-struct<time_trunc(MICROSECOND, TIME '12:34:56.123456'):time(6)>
+struct<time_trunc(MICROSECOND, TIME '12:34:56.123456789'):time(9)>
-- !query output
12:34:56.123456
@@ -1242,7 +1242,7 @@ struct<time_trunc(MICROSECOND, TIME '00:00:00'):time(6)>
-- !query
SELECT time_trunc('HOUR', time'00:00:00.000000001')
-- !query schema
-struct<time_trunc(HOUR, TIME '00:00:00'):time(6)>
+struct<time_trunc(HOUR, TIME '00:00:00.000000001'):time(9)>
-- !query output
00:00:00
@@ -1250,7 +1250,7 @@ struct<time_trunc(HOUR, TIME '00:00:00'):time(6)>
-- !query
SELECT time_trunc('MINUTE', time'00:00:00.000000001')
-- !query schema
-struct<time_trunc(MINUTE, TIME '00:00:00'):time(6)>
+struct<time_trunc(MINUTE, TIME '00:00:00.000000001'):time(9)>
-- !query output
00:00:00
@@ -1258,7 +1258,7 @@ struct<time_trunc(MINUTE, TIME '00:00:00'):time(6)>
-- !query
SELECT time_trunc('SECOND', time'00:00:00.000000001')
-- !query schema
-struct<time_trunc(SECOND, TIME '00:00:00'):time(6)>
+struct<time_trunc(SECOND, TIME '00:00:00.000000001'):time(9)>
-- !query output
00:00:00
@@ -1266,7 +1266,7 @@ struct<time_trunc(SECOND, TIME '00:00:00'):time(6)>
-- !query
SELECT time_trunc('MILLISECOND', time'00:00:00.000000001')
-- !query schema
-struct<time_trunc(MILLISECOND, TIME '00:00:00'):time(6)>
+struct<time_trunc(MILLISECOND, TIME '00:00:00.000000001'):time(9)>
-- !query output
00:00:00
@@ -1274,7 +1274,7 @@ struct<time_trunc(MILLISECOND, TIME '00:00:00'):time(6)>
-- !query
SELECT time_trunc('MICROSECOND', time'00:00:00.000000001')
-- !query schema
-struct<time_trunc(MICROSECOND, TIME '00:00:00'):time(6)>
+struct<time_trunc(MICROSECOND, TIME '00:00:00.000000001'):time(9)>
-- !query output
00:00:00
@@ -1282,7 +1282,7 @@ struct<time_trunc(MICROSECOND, TIME '00:00:00'):time(6)>
-- !query
SELECT time_trunc('HOUR', time'23:59:59.999999999')
-- !query schema
-struct<time_trunc(HOUR, TIME '23:59:59.999999'):time(6)>
+struct<time_trunc(HOUR, TIME '23:59:59.999999999'):time(9)>
-- !query output
23:00:00
@@ -1290,7 +1290,7 @@ struct<time_trunc(HOUR, TIME '23:59:59.999999'):time(6)>
-- !query
SELECT time_trunc('MINUTE', time'23:59:59.999999999')
-- !query schema
-struct<time_trunc(MINUTE, TIME '23:59:59.999999'):time(6)>
+struct<time_trunc(MINUTE, TIME '23:59:59.999999999'):time(9)>
-- !query output
23:59:00
@@ -1298,7 +1298,7 @@ struct<time_trunc(MINUTE, TIME '23:59:59.999999'):time(6)>
-- !query
SELECT time_trunc('SECOND', time'23:59:59.999999999')
-- !query schema
-struct<time_trunc(SECOND, TIME '23:59:59.999999'):time(6)>
+struct<time_trunc(SECOND, TIME '23:59:59.999999999'):time(9)>
-- !query output
23:59:59
@@ -1306,7 +1306,7 @@ struct<time_trunc(SECOND, TIME '23:59:59.999999'):time(6)>
-- !query
SELECT time_trunc('MILLISECOND', time'23:59:59.999999999')
-- !query schema
-struct<time_trunc(MILLISECOND, TIME '23:59:59.999999'):time(6)>
+struct<time_trunc(MILLISECOND, TIME '23:59:59.999999999'):time(9)>
-- !query output
23:59:59.999
@@ -1314,7 +1314,7 @@ struct<time_trunc(MILLISECOND, TIME
'23:59:59.999999'):time(6)>
-- !query
SELECT time_trunc('MICROSECOND', time'23:59:59.999999999')
-- !query schema
-struct<time_trunc(MICROSECOND, TIME '23:59:59.999999'):time(6)>
+struct<time_trunc(MICROSECOND, TIME '23:59:59.999999999'):time(9)>
-- !query output
23:59:59.999999
@@ -1843,7 +1843,7 @@ struct<time_diff(MICROSECOND, TIME '00:00:00', TIME
'12:34:56.123456'):bigint>
-- !query
SELECT time_diff('HOUR', time'00:00:00', time'12:34:56.123456789')
-- !query schema
-struct<time_diff(HOUR, TIME '00:00:00', TIME '12:34:56.123456'):bigint>
+struct<time_diff(HOUR, TIME '00:00:00', TIME '12:34:56.123456789'):bigint>
-- !query output
12
@@ -1851,7 +1851,7 @@ struct<time_diff(HOUR, TIME '00:00:00', TIME
'12:34:56.123456'):bigint>
-- !query
SELECT time_diff('MINUTE', time'00:00:00', time'12:34:56.123456789')
-- !query schema
-struct<time_diff(MINUTE, TIME '00:00:00', TIME '12:34:56.123456'):bigint>
+struct<time_diff(MINUTE, TIME '00:00:00', TIME '12:34:56.123456789'):bigint>
-- !query output
754
@@ -1859,7 +1859,7 @@ struct<time_diff(MINUTE, TIME '00:00:00', TIME
'12:34:56.123456'):bigint>
-- !query
SELECT time_diff('SECOND', time'00:00:00', time'12:34:56.123456789')
-- !query schema
-struct<time_diff(SECOND, TIME '00:00:00', TIME '12:34:56.123456'):bigint>
+struct<time_diff(SECOND, TIME '00:00:00', TIME '12:34:56.123456789'):bigint>
-- !query output
45296
@@ -1867,7 +1867,7 @@ struct<time_diff(SECOND, TIME '00:00:00', TIME
'12:34:56.123456'):bigint>
-- !query
SELECT time_diff('MILLISECOND', time'00:00:00', time'12:34:56.123456789')
-- !query schema
-struct<time_diff(MILLISECOND, TIME '00:00:00', TIME '12:34:56.123456'):bigint>
+struct<time_diff(MILLISECOND, TIME '00:00:00', TIME
'12:34:56.123456789'):bigint>
-- !query output
45296123
@@ -1875,7 +1875,7 @@ struct<time_diff(MILLISECOND, TIME '00:00:00', TIME
'12:34:56.123456'):bigint>
-- !query
SELECT time_diff('MICROSECOND', time'00:00:00', time'12:34:56.123456789')
-- !query schema
-struct<time_diff(MICROSECOND, TIME '00:00:00', TIME '12:34:56.123456'):bigint>
+struct<time_diff(MICROSECOND, TIME '00:00:00', TIME
'12:34:56.123456789'):bigint>
-- !query output
45296123456
@@ -1963,7 +1963,7 @@ struct<time_diff(MICROSECOND, TIME '00:00:00', TIME
'00:00:00'):bigint>
-- !query
SELECT time_diff('HOUR', time'00:00:00', time'00:00:00.000000001')
-- !query schema
-struct<time_diff(HOUR, TIME '00:00:00', TIME '00:00:00'):bigint>
+struct<time_diff(HOUR, TIME '00:00:00', TIME '00:00:00.000000001'):bigint>
-- !query output
0
@@ -1971,7 +1971,7 @@ struct<time_diff(HOUR, TIME '00:00:00', TIME
'00:00:00'):bigint>
-- !query
SELECT time_diff('MINUTE', time'00:00:00', time'00:00:00.000000001')
-- !query schema
-struct<time_diff(MINUTE, TIME '00:00:00', TIME '00:00:00'):bigint>
+struct<time_diff(MINUTE, TIME '00:00:00', TIME '00:00:00.000000001'):bigint>
-- !query output
0
@@ -1979,7 +1979,7 @@ struct<time_diff(MINUTE, TIME '00:00:00', TIME
'00:00:00'):bigint>
-- !query
SELECT time_diff('SECOND', time'00:00:00', time'00:00:00.000000001')
-- !query schema
-struct<time_diff(SECOND, TIME '00:00:00', TIME '00:00:00'):bigint>
+struct<time_diff(SECOND, TIME '00:00:00', TIME '00:00:00.000000001'):bigint>
-- !query output
0
@@ -1987,7 +1987,7 @@ struct<time_diff(SECOND, TIME '00:00:00', TIME
'00:00:00'):bigint>
-- !query
SELECT time_diff('MILLISECOND', time'00:00:00', time'00:00:00.000000001')
-- !query schema
-struct<time_diff(MILLISECOND, TIME '00:00:00', TIME '00:00:00'):bigint>
+struct<time_diff(MILLISECOND, TIME '00:00:00', TIME
'00:00:00.000000001'):bigint>
-- !query output
0
@@ -1995,7 +1995,7 @@ struct<time_diff(MILLISECOND, TIME '00:00:00', TIME
'00:00:00'):bigint>
-- !query
SELECT time_diff('MICROSECOND', time'00:00:00', time'00:00:00.000000001')
-- !query schema
-struct<time_diff(MICROSECOND, TIME '00:00:00', TIME '00:00:00'):bigint>
+struct<time_diff(MICROSECOND, TIME '00:00:00', TIME
'00:00:00.000000001'):bigint>
-- !query output
0
@@ -2003,7 +2003,7 @@ struct<time_diff(MICROSECOND, TIME '00:00:00', TIME
'00:00:00'):bigint>
-- !query
SELECT time_diff('HOUR', time'00:00:00', time'23:59:59.999999999')
-- !query schema
-struct<time_diff(HOUR, TIME '00:00:00', TIME '23:59:59.999999'):bigint>
+struct<time_diff(HOUR, TIME '00:00:00', TIME '23:59:59.999999999'):bigint>
-- !query output
23
@@ -2011,7 +2011,7 @@ struct<time_diff(HOUR, TIME '00:00:00', TIME
'23:59:59.999999'):bigint>
-- !query
SELECT time_diff('MINUTE', time'00:00:00', time'23:59:59.999999999')
-- !query schema
-struct<time_diff(MINUTE, TIME '00:00:00', TIME '23:59:59.999999'):bigint>
+struct<time_diff(MINUTE, TIME '00:00:00', TIME '23:59:59.999999999'):bigint>
-- !query output
1439
@@ -2019,7 +2019,7 @@ struct<time_diff(MINUTE, TIME '00:00:00', TIME
'23:59:59.999999'):bigint>
-- !query
SELECT time_diff('SECOND', time'00:00:00', time'23:59:59.999999999')
-- !query schema
-struct<time_diff(SECOND, TIME '00:00:00', TIME '23:59:59.999999'):bigint>
+struct<time_diff(SECOND, TIME '00:00:00', TIME '23:59:59.999999999'):bigint>
-- !query output
86399
@@ -2027,7 +2027,7 @@ struct<time_diff(SECOND, TIME '00:00:00', TIME
'23:59:59.999999'):bigint>
-- !query
SELECT time_diff('MILLISECOND', time'00:00:00', time'23:59:59.999999999')
-- !query schema
-struct<time_diff(MILLISECOND, TIME '00:00:00', TIME '23:59:59.999999'):bigint>
+struct<time_diff(MILLISECOND, TIME '00:00:00', TIME
'23:59:59.999999999'):bigint>
-- !query output
86399999
@@ -2035,7 +2035,7 @@ struct<time_diff(MILLISECOND, TIME '00:00:00', TIME
'23:59:59.999999'):bigint>
-- !query
SELECT time_diff('MICROSECOND', time'00:00:00', time'23:59:59.999999999')
-- !query schema
-struct<time_diff(MICROSECOND, TIME '00:00:00', TIME '23:59:59.999999'):bigint>
+struct<time_diff(MICROSECOND, TIME '00:00:00', TIME
'23:59:59.999999999'):bigint>
-- !query output
86399999999
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
index a70ef16bdf71..32802a7b7282 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
@@ -929,7 +929,10 @@ class CsvFunctionsSuite extends SharedSparkSession {
(3, LocalTime.of(14, 30, 45, 123000000), "14:30:45.123"),
(4, LocalTime.of(14, 30, 45, 123400000), "14:30:45.1234"),
(5, LocalTime.of(14, 30, 45, 123450000), "14:30:45.12345"),
- (6, LocalTime.of(14, 30, 45, 123456000), "14:30:45.123456")
+ (6, LocalTime.of(14, 30, 45, 123456000), "14:30:45.123456"),
+ (7, LocalTime.of(14, 30, 45, 123456700), "14:30:45.1234567"),
+ (8, LocalTime.of(14, 30, 45, 123456780), "14:30:45.12345678"),
+ (9, LocalTime.of(14, 30, 45, 123456789), "14:30:45.123456789")
)
testData.foreach { case (precision, time, timeStr) =>
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
index 50d5b4be24ee..a5f5a9a0e917 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
@@ -1732,7 +1732,10 @@ class JsonFunctionsSuite extends SharedSparkSession {
(3, LocalTime.of(14, 30, 45, 123000000), "14:30:45.123"),
(4, LocalTime.of(14, 30, 45, 123400000), "14:30:45.1234"),
(5, LocalTime.of(14, 30, 45, 123450000), "14:30:45.12345"),
- (6, LocalTime.of(14, 30, 45, 123456000), "14:30:45.123456")
+ (6, LocalTime.of(14, 30, 45, 123456000), "14:30:45.123456"),
+ (7, LocalTime.of(14, 30, 45, 123456700), "14:30:45.1234567"),
+ (8, LocalTime.of(14, 30, 45, 123456780), "14:30:45.12345678"),
+ (9, LocalTime.of(14, 30, 45, 123456789), "14:30:45.123456789")
)
testData.foreach { case (precision, time, timeStr) =>
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/TimeFunctionsSuiteBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/TimeFunctionsSuiteBase.scala
index bd89dfa45dec..29395eba2dc1 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/TimeFunctionsSuiteBase.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/TimeFunctionsSuiteBase.scala
@@ -87,7 +87,7 @@ abstract class TimeFunctionsSuiteBase extends
SharedSparkSession {
}
test("SPARK-52882: current_time function with specified precision") {
- (0 to 6).foreach { precision: Int =>
+ (0 to TimeType.MAX_PRECISION).foreach { precision: Int =>
// Create a dummy DataFrame with a single row to test the
current_time(precision) function.
val df = spark.range(1)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/XmlFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/XmlFunctionsSuite.scala
index f87bd2df5164..ef875064e0ff 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/XmlFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/XmlFunctionsSuite.scala
@@ -591,7 +591,10 @@ class XmlFunctionsSuite extends SharedSparkSession {
(3, LocalTime.of(14, 30, 45, 123000000), "14:30:45.123"),
(4, LocalTime.of(14, 30, 45, 123400000), "14:30:45.1234"),
(5, LocalTime.of(14, 30, 45, 123450000), "14:30:45.12345"),
- (6, LocalTime.of(14, 30, 45, 123456000), "14:30:45.123456")
+ (6, LocalTime.of(14, 30, 45, 123456000), "14:30:45.123456"),
+ (7, LocalTime.of(14, 30, 45, 123456700), "14:30:45.1234567"),
+ (8, LocalTime.of(14, 30, 45, 123456780), "14:30:45.12345678"),
+ (9, LocalTime.of(14, 30, 45, 123456789), "14:30:45.123456789")
)
testData.foreach { case (precision, time, timeStr) =>
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala
index c4e041e13ebd..909ebf5daf39 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala
@@ -1146,13 +1146,16 @@ abstract class OrcQuerySuite extends OrcQueryTest with
SharedSparkSession {
CAST(TIME'12:34:56.123' AS TIME(3)) as time_p3,
CAST(TIME'12:34:56.1234' AS TIME(4)) as time_p4,
CAST(TIME'12:34:56.12345' AS TIME(5)) as time_p5,
- CAST(TIME'12:34:56.123456' AS TIME(6)) as time_p6
+ CAST(TIME'12:34:56.123456' AS TIME(6)) as time_p6,
+ CAST(TIME'12:34:56.1234567' AS TIME(7)) as time_p7,
+ CAST(TIME'12:34:56.12345678' AS TIME(8)) as time_p8,
+ CAST(TIME'12:34:56.123456789' AS TIME(9)) as time_p9
""")
df.write.mode("overwrite").orc(path)
val result = spark.read.orc(path)
- (0 to 6).foreach { p =>
+ (0 to TimeType.MAX_PRECISION).foreach { p =>
assert(result.schema(s"time_p$p").dataType == TimeType(p))
}
checkAnswer(result, df)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
index 3a06b59b6a18..6b41da841e56 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
@@ -1927,6 +1927,88 @@ class ParquetIOSuite extends ParquetTest with
SharedSparkSession {
}
}
+ test("Read TimeType for the logical TIME(NANOS) type") {
+ val schema = MessageTypeParser.parseMessageType(
+ """message root {
+ | required int64 time_nanos(TIME(NANOS,false));
+ |}""".stripMargin)
+
+ for (dictEnabled <- Seq(true, false)) {
+ withTempDir { dir =>
+ val tablePath = new Path(s"${dir.getCanonicalPath}/times.parquet")
+ val numRecords = 100
+
+ val writer = createParquetWriter(schema, tablePath, dictionaryEnabled
= dictEnabled)
+ (0 until numRecords).foreach { _ =>
+ val record = new SimpleGroup(schema)
+ // Internal storage is nanoseconds since midnight; TIME(NANOS)
writes it unchanged.
+ record.add(0, localTime(23, 59, 59, 123456, 789))
+ writer.write(record)
+ }
+ writer.close
+
+ withAllParquetReaders {
+ val df = spark.read.parquet(tablePath.toString)
+ assertResult(df.schema) {
+ new StructType().add("time_nanos",
TimeType(TimeType.NANOS_PRECISION))
+ }
+ val lt = LocalTime.of(23, 59, 59, 123456789)
+ val expected = (0 until numRecords).map { _ => lt }.toDF()
+ checkAnswer(df, expected)
+ }
+ }
+ }
+ }
+
+ test("TimeType nanosecond round-trip through Parquet") {
+ withAllParquetReaders {
+ Seq(7, 8, 9).foreach { p =>
+ withTempPath { path =>
+ val df = spark.sql(
+ s"SELECT CAST(TIME '23:59:59.123456789' AS TIME($p)) AS t")
+ df.write.parquet(path.getCanonicalPath)
+ val readBack = spark.read.parquet(path.getCanonicalPath)
+ assert(readBack.schema.head.dataType === TimeType(p))
+ checkAnswer(readBack, df)
+ }
+ }
+ }
+ }
+
+ test("Read TIME with a lower precision than the stored value truncates in
both readers") {
+ // A raw TIME(NANOS) file carrying full nanosecond precision (.123456789),
read back with an
+ // explicit lower precision (TIME(7)). Both the vectorized and the
row-based reader must drop
+ // the sub-100ns digits (.123456789 -> .1234567); otherwise the requested
type's precision is
+ // violated. Covers both the dictionary and plain decode paths.
+ val schema = MessageTypeParser.parseMessageType(
+ """message root {
+ | required int64 time_nanos(TIME(NANOS,false));
+ |}""".stripMargin)
+ val readSchema = new StructType().add("time_nanos", TimeType(7))
+ val expected = LocalTime.of(23, 59, 59, 123456700)
+
+ for (dictEnabled <- Seq(true, false)) {
+ withTempDir { dir =>
+ val tablePath = new Path(s"${dir.getCanonicalPath}/times.parquet")
+ val numRecords = 100
+
+ val writer = createParquetWriter(schema, tablePath, dictionaryEnabled
= dictEnabled)
+ (0 until numRecords).foreach { _ =>
+ val record = new SimpleGroup(schema)
+ record.add(0, localTime(23, 59, 59, 123456, 789))
+ writer.write(record)
+ }
+ writer.close
+
+ withAllParquetReaders {
+ val df = spark.read.schema(readSchema).parquet(tablePath.toString)
+ assertResult(df.schema)(readSchema)
+ checkAnswer(df, (0 until numRecords).map(_ => Row(expected)))
+ }
+ }
+ }
+ }
+
// Deterministic INT32 sample shared by the INT32 widening tests below.
Mixes sign,
// zero, and MIN/MAX boundaries to catch sign-extension and precision
regressions.
private def widenSampleAt(i: Int): Int = i % 5 match {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimeTypeParquetOpsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimeTypeParquetOpsSuite.scala
index 36221cec8b2c..fa44297103ea 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimeTypeParquetOpsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimeTypeParquetOpsSuite.scala
@@ -28,26 +28,25 @@ import org.apache.spark.sql.types.TimeType
/**
* Unit tests for [[TimeTypeParquetOps.requireCompatibleParquetType]].
*
- * TimeType is written to Parquet as INT64 TIME(MICROS,
isAdjustedToUTC=false). The
- * read-path guard accepts any INT64 TIME(MICROS) column - both
isAdjustedToUTC values -
- * and rejects every other primitive/annotation combination so that reading
fails loudly
- * rather than silently mis-decoding (e.g. interpreting NANOS as MICROS, which
would be
- * off by 1000x).
+ * TimeType is stored in Parquet as INT64 TIME(MICROS, isAdjustedToUTC=false)
for precision
+ * 0..6 and INT64 TIME(NANOS, isAdjustedToUTC=false) for precision 7..9. The
read-path guard
+ * accepts both of those local-time encodings and rejects every other
primitive/annotation
+ * combination so that reading fails loudly rather than silently mis-decoding.
*
* SPARK-57416: the guard accepts isAdjustedToUTC=true to mirror the legacy
- * ParquetRowConverter guard (which only checks the TIME annotation and the
MICROS unit).
- * Spark's TimeType is zone-less local time, so the flag carries no extra
information on
- * read and the raw micros-of-day value decodes identically either way. This
keeps the
- * framework read path consistent with both the legacy row-based reader and
the vectorized
- * reader.
+ * ParquetRowConverter guard (which only checks the TIME annotation and unit).
Spark's
+ * TimeType is zone-less local time, so the flag carries no extra information
on read and the
+ * raw time-of-day value decodes identically either way. This keeps the
framework read path
+ * consistent with both the legacy row-based reader and the vectorized reader.
*/
class TimeTypeParquetOpsSuite extends SparkFunSuite {
private val timeMicros = TimeType(TimeType.MICROS_PRECISION)
+ private val timeNanos = TimeType(TimeType.NANOS_PRECISION)
// ---------- accept ----------
- test("accepts INT64 TIME(MICROS, isAdjustedToUTC=false) - the canonical
encoding") {
+ test("accepts INT64 TIME(MICROS, isAdjustedToUTC=false) - the canonical
micros encoding") {
val field = Types.primitive(INT64, REQUIRED)
.as(LogicalTypeAnnotation.timeType(false, TimeUnit.MICROS))
.named("c")
@@ -68,17 +67,29 @@ class TimeTypeParquetOpsSuite extends SparkFunSuite {
TimeTypeParquetOps.requireCompatibleParquetType(timeMicros, field)
}
- // ---------- the primary reject paths ----------
-
- test("rejects raw INT64 with no logical type annotation") {
- val field = Types.primitive(INT64, REQUIRED).named("c")
- assertRejects(timeMicros, field)
+ test("accepts INT64 TIME(NANOS, isAdjustedToUTC=false) - the canonical nanos
encoding") {
+ val field = Types.primitive(INT64, REQUIRED)
+ .as(LogicalTypeAnnotation.timeType(false, TimeUnit.NANOS))
+ .named("c")
+ // Must not throw.
+ TimeTypeParquetOps.requireCompatibleParquetType(timeNanos, field)
}
- test("rejects INT64 TIME(NANOS, isAdjustedToUTC=false)") {
+ test("accepts INT64 TIME(NANOS, isAdjustedToUTC=true) - matches legacy
lenient read") {
+ // Same zone-less reasoning as the MICROS case (SPARK-57416): the
UTC-adjustment flag
+ // carries no information for Spark's local-time TimeType, so a NANOS
column is accepted
+ // regardless of the flag.
val field = Types.primitive(INT64, REQUIRED)
- .as(LogicalTypeAnnotation.timeType(false, TimeUnit.NANOS))
+ .as(LogicalTypeAnnotation.timeType(true, TimeUnit.NANOS))
.named("c")
+ // Must not throw.
+ TimeTypeParquetOps.requireCompatibleParquetType(timeNanos, field)
+ }
+
+ // ---------- the primary reject paths ----------
+
+ test("rejects raw INT64 with no logical type annotation") {
+ val field = Types.primitive(INT64, REQUIRED).named("c")
assertRejects(timeMicros, field)
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/sources/PartitionedWriteSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/sources/PartitionedWriteSuite.scala
index 5d9a7b0aafc7..fb37ffaa7aa2 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/sources/PartitionedWriteSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/sources/PartitionedWriteSuite.scala
@@ -232,7 +232,10 @@ class PartitionedWriteSuite extends SharedSparkSession {
"00:01:02.999999" -> TimeType(6),
"12:00:00" -> TimeType(1),
"23:59:59.000001" -> TimeType(),
- "23:59:59.999999" -> TimeType(6)
+ "23:59:59.999999" -> TimeType(6),
+ "00:00:00.0000019" -> TimeType(7),
+ "12:34:56.12345678" -> TimeType(8),
+ "23:59:59.999999999" -> TimeType(9)
).foreach { case (timeStr, timeType) =>
withTempPath { f =>
val df = sql(s"select 0 AS id, cast('$timeStr' as ${timeType.sql}) AS
tt")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
