This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f3d1de5ced5a [SPARK-57372][SQL][FOLLOWUP] Remove dead TimeType code
paths after Types Framework flag removal
f3d1de5ced5a is described below
commit f3d1de5ced5a2869c0b64bae032ea7d7aa8ee8db
Author: David Milicevic <[email protected]>
AuthorDate: Wed Jun 17 11:02:34 2026 -0700
[SPARK-57372][SQL][FOLLOWUP] Remove dead TimeType code paths after Types
Framework flag removal
### What changes were proposed in this pull request?
Follow-up to SPARK-57372 (#56439). That PR removed
`spark.sql.types.framework.enabled` and made the Types Framework the
unconditional integration path for `TimeType`, so `TypeApiOps.apply` /
`TypeOps.apply` / `ConnectTypeOps` now always return `Some(...)` for
`TimeType`. As a result, the legacy `TimeType` arms that remained in the
`...Default` fallbacks behind those dispatches
(`Ops(dt).map(_.foo).getOrElse(fooDefault(dt))`) are unreachable for `TimeType`.
This removes that dead code throughout the framework's integration points
-- physical type, row accessors, literals, encoders, type conversion, the
interpreted cast-to-string path, Row JSON, Hive result, Python conversion,
Arrow (both directions), the Spark Connect proto/Arrow/literal converters, and
the Thrift type-id mapping -- along with the imports and flag-era comments that
only those arms needed.
Arms that are still reachable are deliberately left in place:
- the cast-to-string **codegen** path (`ToStringBase.castToStringCode`) is
a direct match, not a framework `getOrElse`, so its `TimeType` branch stays;
- `ArrowWriter.TimeWriter` remains -- it is what
`TimeTypeOps.createArrowFieldWriter` instantiates;
- `CatalystTypeConverters.TimeConverter` remains -- it is used by the
value-keyed `convertToCatalyst`;
- the class/encoder reverse-lookups and all nanosecond-timestamp arms are
untouched.
### Why are the changes needed?
These `TimeType` branches can no longer execute after the flag removal, so
they are dead code: redundant with the framework `Ops` implementations and a
hazard for future readers who might assume they are still reached. Removing
them collapses each integration point to the single framework path the flag
removal established.
### Does this PR introduce _any_ user-facing change?
No. This is a dead-code removal; the framework path produces results
identical to the removed (unreachable) arms, and `spark.sql.timeType.enabled`
and the other feature gates are unaffected.
### How was this patch tested?
Pure dead-code removal, verified behavior-preserving by existing suites
that already run with the framework unconditionally enabled. The following all
pass: `TimeTypeOpsSuite`, `TimeExpressionsSuite`, `TimeFormatterSuite`,
`RowJsonSuite`, `CatalystTypeConvertersSuite`, `RowEncoderSuite`,
`ExpressionEncoderSuite`, `LiteralExpressionSuite`, `ArrowUtilsSuite`,
`ArrowWriterSuite`, `HiveResultSuite`, `SparkExecuteStatementOperationSuite`,
and the Spark Connect `ArrowEncoderSuite`. The `tim [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8 (claude-opus-4-8)
Closes #56566 from davidm-db/time-deadcode-cleanup.
Authored-by: David Milicevic <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
sql/api/src/main/scala/org/apache/spark/sql/Row.scala | 11 ++---------
.../spark/sql/catalyst/encoders/RowEncoder.scala | 3 +--
.../scala/org/apache/spark/sql/util/ArrowUtils.scala | 3 ---
.../spark/sql/catalyst/CatalystTypeConverters.scala | 1 -
.../org/apache/spark/sql/catalyst/InternalRow.scala | 2 +-
.../spark/sql/catalyst/encoders/EncoderUtils.scala | 3 +--
.../sql/catalyst/expressions/SpecificInternalRow.scala | 2 +-
.../spark/sql/catalyst/expressions/ToStringBase.scala | 2 --
.../catalyst/expressions/codegen/CodeGenerator.scala | 2 +-
.../spark/sql/catalyst/expressions/literals.scala | 1 -
.../spark/sql/catalyst/types/PhysicalDataType.scala | 3 +--
.../apache/spark/sql/execution/arrow/ArrowWriter.scala | 1 -
.../sql/connect/client/arrow/ArrowDeserializer.scala | 5 -----
.../sql/connect/client/arrow/ArrowSerializer.scala | 8 +-------
.../sql/connect/common/DataTypeProtoConverter.scala | 13 -------------
.../connect/common/LiteralValueProtoConverter.scala | 18 ------------------
.../org/apache/spark/sql/execution/HiveResult.scala | 1 -
.../spark/sql/execution/python/EvaluatePython.scala | 4 ++--
.../thriftserver/SparkExecuteStatementOperation.scala | 1 -
19 files changed, 11 insertions(+), 73 deletions(-)
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/Row.scala
b/sql/api/src/main/scala/org/apache/spark/sql/Row.scala
index b6b5c7da9644..137a8863a604 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/Row.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/Row.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql
import java.sql.{Date, Timestamp}
-import java.time.{Instant, LocalDate, LocalDateTime, LocalTime}
+import java.time.{Instant, LocalDate, LocalDateTime}
import java.util.Base64
import scala.collection.mutable
@@ -32,7 +32,7 @@ import org.json4s.jackson.JsonMethods.{compact, pretty,
render}
import org.apache.spark.SparkIllegalArgumentException
import org.apache.spark.annotation.{Stable, Unstable}
import org.apache.spark.sql.catalyst.expressions.GenericRow
-import org.apache.spark.sql.catalyst.util.{DateFormatter, SparkDateTimeUtils,
TimeFormatter, TimestampFormatter, UDTUtils}
+import org.apache.spark.sql.catalyst.util.{DateFormatter, SparkDateTimeUtils,
TimestampFormatter, UDTUtils}
import org.apache.spark.sql.errors.DataTypeErrors
import org.apache.spark.sql.errors.DataTypeErrors.{toSQLType, toSQLValue}
import org.apache.spark.sql.internal.SqlApiConf
@@ -620,7 +620,6 @@ trait Row extends Serializable {
lazy val zoneId =
SparkDateTimeUtils.getZoneId(SqlApiConf.get.sessionLocalTimeZone)
lazy val dateFormatter = DateFormatter()
lazy val timestampFormatter = TimestampFormatter(zoneId)
- lazy val timeFormatter = TimeFormatter.getFractionFormatter()
// Convert an iterator of values to a json array
def iteratorToJsonArray(iterator: Iterator[_], elementType: DataType):
JArray = {
@@ -648,12 +647,6 @@ trait Row extends Serializable {
case (b: Byte, _) => JLong(b)
case (s: Short, _) => JLong(s)
case (i: Int, _) => JLong(i)
- // A public Row holds the external java.time.LocalTime for a TimeType
column (SPARK-54451).
- // With the Types Framework off, TypeApiOps returns None and we land
here, so render the
- // LocalTime directly, mirroring HiveResult's legacy fallback. The Long
case below is the
- // internal representation that a public Row never holds; it stays for
non-Row callers.
- case (lt: LocalTime, _: TimeType) => JString(timeFormatter.format(lt))
- case (nanos: Long, _: TimeType) => JString(timeFormatter.format(nanos))
case (l: Long, _) => JLong(l)
case (f: Float, _) => JDouble(f)
case (d: Double, _) => JDouble(d)
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
index 2ad579b2cca5..69b5cc53fa76 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
@@ -21,7 +21,7 @@ import scala.collection.mutable
import scala.reflect.classTag
import org.apache.spark.sql.{AnalysisException, Row}
-import org.apache.spark.sql.catalyst.encoders.AgnosticEncoders.{BinaryEncoder,
BoxedBooleanEncoder, BoxedByteEncoder, BoxedDoubleEncoder, BoxedFloatEncoder,
BoxedIntEncoder, BoxedLongEncoder, BoxedShortEncoder, CalendarIntervalEncoder,
CharEncoder, DateEncoder, DayTimeIntervalEncoder, EncoderField,
GeographyEncoder, GeometryEncoder, InstantEncoder, IterableEncoder,
JavaDecimalEncoder, LocalDateEncoder, LocalDateTimeEncoder, LocalTimeEncoder,
MapEncoder, NullEncoder, RowEncoder => Agnosti [...]
+import org.apache.spark.sql.catalyst.encoders.AgnosticEncoders.{BinaryEncoder,
BoxedBooleanEncoder, BoxedByteEncoder, BoxedDoubleEncoder, BoxedFloatEncoder,
BoxedIntEncoder, BoxedLongEncoder, BoxedShortEncoder, CalendarIntervalEncoder,
CharEncoder, DateEncoder, DayTimeIntervalEncoder, EncoderField,
GeographyEncoder, GeometryEncoder, InstantEncoder, IterableEncoder,
JavaDecimalEncoder, LocalDateEncoder, LocalDateTimeEncoder, MapEncoder,
NullEncoder, RowEncoder => AgnosticRowEncoder, Strin [...]
import org.apache.spark.sql.errors.DataTypeErrorsBase
import org.apache.spark.sql.internal.SqlApiConf
import org.apache.spark.sql.types._
@@ -101,7 +101,6 @@ object RowEncoder extends DataTypeErrorsBase {
case TimestampNTZType => LocalDateTimeEncoder
case DateType if SqlApiConf.get.datetimeJava8ApiEnabled =>
LocalDateEncoder(lenient)
case DateType => DateEncoder(lenient)
- case _: TimeType => LocalTimeEncoder
case CalendarIntervalType => CalendarIntervalEncoder
case _: DayTimeIntervalType => DayTimeIntervalEncoder
case _: YearMonthIntervalType => YearMonthIntervalEncoder
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala
b/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala
index 1c1024fc0152..b0980a911fdd 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala
@@ -67,7 +67,6 @@ private[sql] object ArrowUtils {
case TimestampType => new ArrowType.Timestamp(TimeUnit.MICROSECOND,
timeZoneId)
case TimestampNTZType =>
new ArrowType.Timestamp(TimeUnit.MICROSECOND, null)
- case _: TimeType => new ArrowType.Time(TimeUnit.NANOSECOND, 8 * 8)
case NullType => ArrowType.Null.INSTANCE
case _: YearMonthIntervalType => new
ArrowType.Interval(IntervalUnit.YEAR_MONTH)
case _: DayTimeIntervalType => new
ArrowType.Duration(TimeUnit.MICROSECOND)
@@ -101,8 +100,6 @@ private[sql] object ArrowUtils {
if ts.getUnit == TimeUnit.MICROSECOND && ts.getTimezone == null =>
TimestampNTZType
case ts: ArrowType.Timestamp if ts.getUnit == TimeUnit.MICROSECOND =>
TimestampType
- case t: ArrowType.Time if t.getUnit == TimeUnit.NANOSECOND &&
t.getBitWidth == 8 * 8 =>
- TimeType(TimeType.MICROS_PRECISION)
case ArrowType.Null.INSTANCE => NullType
case yi: ArrowType.Interval if yi.getUnit == IntervalUnit.YEAR_MONTH =>
YearMonthIntervalType()
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
index 3924337a65bd..97804d1e8463 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
@@ -84,7 +84,6 @@ object CatalystTypeConverters {
new GeometryConverter(g)
case DateType if SQLConf.get.datetimeJava8ApiEnabled =>
LocalDateConverter
case DateType => DateConverter
- case _: TimeType => TimeConverter
case TimestampType if SQLConf.get.datetimeJava8ApiEnabled =>
InstantConverter
case TimestampType => TimestampConverter
case TimestampNTZType => TimestampNTZConverter
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala
index abec18230973..1cb1a0b5ff23 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala
@@ -196,7 +196,7 @@ object InternalRow {
case ShortType => (input, v) => input.setShort(ordinal,
v.asInstanceOf[Short])
case IntegerType | DateType | _: YearMonthIntervalType =>
(input, v) => input.setInt(ordinal, v.asInstanceOf[Int])
- case LongType | TimestampType | TimestampNTZType | _: DayTimeIntervalType
| _: TimeType =>
+ case LongType | TimestampType | TimestampNTZType | _: DayTimeIntervalType
=>
(input, v) => input.setLong(ordinal, v.asInstanceOf[Long])
case FloatType => (input, v) => input.setFloat(ordinal,
v.asInstanceOf[Float])
case DoubleType => (input, v) => input.setDouble(ordinal,
v.asInstanceOf[Double])
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/EncoderUtils.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/EncoderUtils.scala
index a4acc6e20d70..dbd632bb7286 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/EncoderUtils.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/EncoderUtils.scala
@@ -25,7 +25,7 @@ import org.apache.spark.sql.catalyst.expressions.Expression
import org.apache.spark.sql.catalyst.types.{PhysicalBinaryType,
PhysicalIntegerType, PhysicalLongType}
import org.apache.spark.sql.catalyst.types.ops.TypeOps
import org.apache.spark.sql.catalyst.util.{ArrayData, MapData}
-import org.apache.spark.sql.types.{ArrayType, BinaryType, BooleanType,
ByteType, CalendarIntervalType, DataType, DateType, DayTimeIntervalType,
Decimal, DecimalType, DoubleType, FloatType, GeographyType, GeometryType,
IntegerType, LongType, MapType, ObjectType, ShortType, StringType, StructType,
TimestampNTZType, TimestampType, TimeType, UserDefinedType, VariantType,
YearMonthIntervalType}
+import org.apache.spark.sql.types.{ArrayType, BinaryType, BooleanType,
ByteType, CalendarIntervalType, DataType, DateType, DayTimeIntervalType,
Decimal, DecimalType, DoubleType, FloatType, GeographyType, GeometryType,
IntegerType, LongType, MapType, ObjectType, ShortType, StringType, StructType,
TimestampNTZType, TimestampType, UserDefinedType, VariantType,
YearMonthIntervalType}
import org.apache.spark.unsafe.types.{BinaryView, CalendarInterval,
UTF8String, VariantVal}
/**
@@ -106,7 +106,6 @@ object EncoderUtils {
case _: DecimalType => classOf[Decimal]
case _: DayTimeIntervalType => classOf[PhysicalLongType.InternalType]
case _: YearMonthIntervalType =>
classOf[PhysicalIntegerType.InternalType]
- case _: TimeType => classOf[PhysicalLongType.InternalType]
case _: StringType => classOf[UTF8String]
case _: StructType => classOf[InternalRow]
case _: ArrayType => classOf[ArrayData]
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SpecificInternalRow.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SpecificInternalRow.scala
index 88c3e181e645..b92a9d8f90df 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SpecificInternalRow.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SpecificInternalRow.scala
@@ -219,7 +219,7 @@ final class SpecificInternalRow(val values:
Array[MutableValue]) extends BaseGen
// We use INT for DATE and YearMonthIntervalType internally
case IntegerType | DateType | _: YearMonthIntervalType => new MutableInt
// We use Long for Timestamp, Timestamp without time zone and
DayTimeInterval internally
- case LongType | TimestampType | TimestampNTZType | _: DayTimeIntervalType
| _: TimeType =>
+ case LongType | TimestampType | TimestampNTZType | _: DayTimeIntervalType
=>
new MutableLong
case FloatType => new MutableFloat
case DoubleType => new MutableDouble
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
index 56e494ced8fe..b8d389259544 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
@@ -85,8 +85,6 @@ trait ToStringBase { self: UnaryExpression with
TimeZoneAwareExpression =>
acceptAny[Long](t => UTF8String.fromString(timestampFormatter.format(t)))
case TimestampNTZType =>
acceptAny[Long](t =>
UTF8String.fromString(timestampNTZFormatter.format(t)))
- case _: TimeType =>
- acceptAny[Long](t => UTF8String.fromString(timeFormatter.format(t)))
case ArrayType(et, _) =>
acceptAny[ArrayData](array => {
val builder = new UTF8StringBuilder
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
index 2dfb590197b0..def7ec39b571 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
@@ -2057,7 +2057,7 @@ object CodeGenerator extends Logging {
case ByteType => java.lang.Byte.TYPE
case ShortType => java.lang.Short.TYPE
case IntegerType | DateType | _: YearMonthIntervalType =>
java.lang.Integer.TYPE
- case LongType | TimestampType | TimestampNTZType | _: DayTimeIntervalType
| _: TimeType =>
+ case LongType | TimestampType | TimestampNTZType | _: DayTimeIntervalType
=>
java.lang.Long.TYPE
case FloatType => java.lang.Float.TYPE
case DoubleType => java.lang.Double.TYPE
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
index 75526db24ddf..b1ed1a494be8 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
@@ -227,7 +227,6 @@ object Literal {
case DateType => create(0, DateType)
case TimestampType => create(0L, TimestampType)
case TimestampNTZType => create(0L, TimestampNTZType)
- case t: TimeType => create(0L, t)
case it: DayTimeIntervalType => create(0L, it)
case it: YearMonthIntervalType => create(0, it)
case c: CharType =>
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/PhysicalDataType.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/PhysicalDataType.scala
index f682d30235f1..9acbf4be66f9 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/PhysicalDataType.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/PhysicalDataType.scala
@@ -24,7 +24,7 @@ import org.apache.spark.sql.catalyst.expressions.{Ascending,
BoundReference, Int
import org.apache.spark.sql.catalyst.types.ops.TypeOps
import org.apache.spark.sql.catalyst.util.{ArrayData, CollationFactory,
MapData, SQLOrderingUtil}
import org.apache.spark.sql.errors.QueryExecutionErrors
-import org.apache.spark.sql.types.{ArrayType, BinaryType, BooleanType,
ByteExactNumeric, ByteType, CalendarIntervalType, CharType, DataType, DateType,
DayTimeIntervalType, Decimal, DecimalExactNumeric, DecimalType,
DoubleExactNumeric, DoubleType, FloatExactNumeric, FloatType, FractionalType,
GeographyType, GeometryType, IntegerExactNumeric, IntegerType, IntegralType,
LongExactNumeric, LongType, MapType, NullType, NumericType, ShortExactNumeric,
ShortType, StringType, StructField, StructT [...]
+import org.apache.spark.sql.types.{ArrayType, BinaryType, BooleanType,
ByteExactNumeric, ByteType, CalendarIntervalType, CharType, DataType, DateType,
DayTimeIntervalType, Decimal, DecimalExactNumeric, DecimalType,
DoubleExactNumeric, DoubleType, FloatExactNumeric, FloatType, FractionalType,
GeographyType, GeometryType, IntegerExactNumeric, IntegerType, IntegralType,
LongExactNumeric, LongType, MapType, NullType, NumericType, ShortExactNumeric,
ShortType, StringType, StructField, StructT [...]
import org.apache.spark.unsafe.types.{BinaryView, ByteArray,
TimestampNanosVal, UTF8String, VariantVal}
import org.apache.spark.util.ArrayImplicits._
@@ -58,7 +58,6 @@ object PhysicalDataType {
case DayTimeIntervalType(_, _) => PhysicalLongType
case YearMonthIntervalType(_, _) => PhysicalIntegerType
case DateType => PhysicalIntegerType
- case _: TimeType => PhysicalLongType
case ArrayType(elementType, containsNull) =>
PhysicalArrayType(elementType, containsNull)
case StructType(fields) => PhysicalStructType(fields)
case MapType(keyType, valueType, valueContainsNull) =>
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala
index df5ea031e79f..1e2ae058ac63 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala
@@ -77,7 +77,6 @@ object ArrowWriter {
case (DateType, vector: DateDayVector) => new DateWriter(vector)
case (TimestampType, vector: TimeStampMicroTZVector) => new
TimestampWriter(vector)
case (TimestampNTZType, vector: TimeStampMicroVector) => new
TimestampNTZWriter(vector)
- case (_: TimeType, vector: TimeNanoVector) => new TimeWriter(vector)
case (ArrayType(_, _), vector: ListVector) =>
val elementVector = createFieldWriter(vector.getDataVector())
new ArrayWriter(vector, elementVector)
diff --git
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowDeserializer.scala
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowDeserializer.scala
index f2786c61d1b5..ad8e53e32e0c 100644
---
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowDeserializer.scala
+++
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowDeserializer.scala
@@ -211,11 +211,6 @@ object ArrowDeserializers {
new LeafFieldDeserializer[LocalDateTime](encoder, v, timeZoneId) {
override def value(i: Int): LocalDateTime =
reader.getLocalDateTime(i)
}
- case (LocalTimeEncoder, v: FieldVector) =>
- new LeafFieldDeserializer[LocalTime](encoder, v, timeZoneId) {
- override def value(i: Int): LocalTime = reader.getLocalTime(i)
- }
-
case (OptionEncoder(value), v) =>
val deserializer = deserializerFor(value, v, timeZoneId)
new Deserializer[Any] {
diff --git
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala
index 786d6a1d3bbb..02918e4eb140 100644
---
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala
+++
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala
@@ -20,7 +20,7 @@ import java.io.{ByteArrayOutputStream, OutputStream}
import java.lang.invoke.{MethodHandles, MethodType}
import java.math.{BigDecimal => JBigDecimal, BigInteger => JBigInteger}
import java.nio.channels.Channels
-import java.time.{Duration, Instant, LocalDate, LocalDateTime, LocalTime,
Period}
+import java.time.{Duration, Instant, LocalDate, LocalDateTime, Period}
import java.util.{Map => JMap, Objects}
import scala.jdk.CollectionConverters._
@@ -398,12 +398,6 @@ object ArrowSerializer {
override def set(index: Int, value: LocalDateTime): Unit =
vector.setSafe(index,
SparkDateTimeUtils.localDateTimeToMicros(value))
}
- case (LocalTimeEncoder, v: TimeNanoVector) =>
- new FieldSerializer[LocalTime, TimeNanoVector](v) {
- override def set(index: Int, value: LocalTime): Unit =
- vector.setSafe(index, SparkDateTimeUtils.localTimeToNanos(value))
- }
-
case (OptionEncoder(value), v) =>
new Serializer {
private[this] val delegate: Serializer = serializerFor(value, v)
diff --git
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala
index 11b1b394b1ed..10ea4b3c4521 100644
---
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala
+++
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala
@@ -57,13 +57,6 @@ object DataTypeProtoConverter {
case proto.DataType.KindCase.DATE => DateType
case proto.DataType.KindCase.TIMESTAMP => TimestampType
case proto.DataType.KindCase.TIMESTAMP_NTZ => TimestampNTZType
- case proto.DataType.KindCase.TIME =>
- if (t.getTime.hasPrecision) {
- TimeType(t.getTime.getPrecision)
- } else {
- TimeType()
- }
-
case proto.DataType.KindCase.CALENDAR_INTERVAL => CalendarIntervalType
case proto.DataType.KindCase.YEAR_MONTH_INTERVAL =>
toCatalystYearMonthIntervalType(t.getYearMonthInterval)
@@ -241,12 +234,6 @@ object DataTypeProtoConverter {
case TimestampNTZType => ProtoDataTypes.TimestampNTZType
- case TimeType(precision) =>
- proto.DataType
- .newBuilder()
-
.setTime(proto.DataType.Time.newBuilder().setPrecision(precision).build())
- .build()
-
case CalendarIntervalType => ProtoDataTypes.CalendarIntervalType
case YearMonthIntervalType(startField, endField) =>
diff --git
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/LiteralValueProtoConverter.scala
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/LiteralValueProtoConverter.scala
index 29623c7cbdb9..33da07cc1b5b 100644
---
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/LiteralValueProtoConverter.scala
+++
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/LiteralValueProtoConverter.scala
@@ -118,11 +118,6 @@ object LiteralValueProtoConverter {
case v: Date => builder.setDate(SparkDateTimeUtils.fromJavaDate(v))
case v: Duration =>
builder.setDayTimeInterval(SparkIntervalUtils.durationToMicros(v))
case v: Period =>
builder.setYearMonthInterval(SparkIntervalUtils.periodToMonths(v))
- case v: LocalTime =>
- builder.setTime(
- builder.getTimeBuilder
- .setNano(SparkDateTimeUtils.localTimeToNanos(v))
- .setPrecision(TimeType.DEFAULT_PRECISION))
case v: Array[_] => builder.setArray(arrayBuilder(v))
case v: CalendarInterval =>
builder.setCalendarInterval(calendarIntervalBuilder(v.months, v.days,
v.microseconds))
@@ -235,11 +230,6 @@ object LiteralValueProtoConverter {
builder.setMap(mapBuilder(v, keyType, valueType))
case (v, structType: StructType) =>
builder.setStruct(structBuilder(v, structType))
- case (v: LocalTime, timeType: TimeType) =>
- builder.setTime(
- builder.getTimeBuilder
- .setNano(SparkDateTimeUtils.localTimeToNanos(v))
- .setPrecision(timeType.precision))
case _ => toLiteralProtoBuilderInternal(literal, options)
}
@@ -430,8 +420,6 @@ object LiteralValueProtoConverter {
v => SparkIntervalUtils.microsToDuration(v.getDayTimeInterval)
case proto.DataType.KindCase.YEAR_MONTH_INTERVAL =>
v => SparkIntervalUtils.monthsToPeriod(v.getYearMonthInterval)
- case proto.DataType.KindCase.TIME =>
- v => SparkDateTimeUtils.nanosToLocalTime(v.getTime.getNano)
case proto.DataType.KindCase.DECIMAL => v =>
Decimal(v.getDecimal.getValue)
case proto.DataType.KindCase.CALENDAR_INTERVAL =>
v =>
@@ -594,12 +582,6 @@ object LiteralValueProtoConverter {
builder.setYearMonthInterval(proto.DataType.YearMonthInterval.newBuilder().build())
case proto.Expression.Literal.LiteralTypeCase.DAY_TIME_INTERVAL =>
builder.setDayTimeInterval(proto.DataType.DayTimeInterval.newBuilder().build())
- case proto.Expression.Literal.LiteralTypeCase.TIME =>
- val timeBuilder = proto.DataType.Time.newBuilder()
- if (literal.getTime.hasPrecision) {
- timeBuilder.setPrecision(literal.getTime.getPrecision)
- }
- builder.setTime(timeBuilder.build())
case proto.Expression.Literal.LiteralTypeCase.ARRAY =>
if (literal.getArray.hasElementType) {
builder.setArray(
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
index 671451087435..c481f4eb7c11 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
@@ -127,7 +127,6 @@ object HiveResult extends SQLConfHelper {
case (b, BooleanType) => b.toString
case (d: Date, DateType) => formatters.date.format(d)
case (ld: LocalDate, DateType) => formatters.date.format(ld)
- case (lt: LocalTime, _: TimeType) => formatters.time.format(lt)
case (t: Timestamp, TimestampType) => formatters.timestamp.format(t)
case (i: Instant, TimestampType) => formatters.timestamp.format(i)
case (l: LocalDateTime, TimestampNTZType) => formatters.timestamp.format(l)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala
index adee0b2ea19a..874c480b4c9f 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala
@@ -48,7 +48,7 @@ object EvaluatePython {
private def needConversionInPythonDefault(dt: DataType): Boolean = dt match {
case DateType | TimestampType | TimestampNTZType | VariantType | _:
DayTimeIntervalType
- | _: TimeType | _: GeometryType | _: GeographyType => true
+ | _: GeometryType | _: GeographyType => true
case _: StructType => true
case _: UserDefinedType[_] => true
case ArrayType(elementType, _) => needConversionInPython(elementType)
@@ -170,7 +170,7 @@ object EvaluatePython {
case c: Int => c
}
- case TimestampType | TimestampNTZType | _: DayTimeIntervalType | _:
TimeType => (obj: Any) =>
+ case TimestampType | TimestampNTZType | _: DayTimeIntervalType => (obj:
Any) =>
nullSafeConvert(obj) {
case c: Long => c
// Py4J serializes values between MIN_INT and MAX_INT as Ints, not
Longs
diff --git
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
index 46302b316b75..8757fad2933c 100644
---
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
+++
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
@@ -347,7 +347,6 @@ object SparkExecuteStatementOperation {
case _: StringType => TTypeId.STRING_TYPE
case _: DecimalType => TTypeId.DECIMAL_TYPE
case DateType => TTypeId.DATE_TYPE
- case _: TimeType => TTypeId.STRING_TYPE
// TODO: Shall use TIMESTAMPLOCALTZ_TYPE, keep AS-IS now for
// unnecessary behavior change
case TimestampType => TTypeId.TIMESTAMP_TYPE
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
