peter-toth commented on code in PR #56158:
URL: https://github.com/apache/spark/pull/56158#discussion_r3330289473
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala:
##########
@@ -97,6 +99,14 @@ object RowEncoder extends DataTypeErrorsBase {
case TimestampType if SqlApiConf.get.datetimeJava8ApiEnabled =>
InstantEncoder(lenient)
case TimestampType => TimestampEncoder(lenient)
case TimestampNTZType => LocalDateTimeEncoder
+ // Nano timestamp types intentionally do not honor `lenient`: legacy
`java.sql.Timestamp` /
+ // `java.sql.Date` external types are out of scope for nanosecond
precision (SPARK-57033).
+ case t: TimestampNTZNanosType =>
Review Comment:
The PR's `DatasetSuite` / `JavaDatasetSuite` tests confirm
`createDataFrame(rows, schemaWithNanos).collect()` works under classic mode.
The same call under Spark Connect throws before it ever reaches `collect()`:
- `sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:51-77`
(`toArrowTypeDefault`) has no `TimestampNTZNanosType` / `TimestampLTZNanosType`
case — falls to `case _ => throw ExecutionErrors.unsupportedDataTypeError(dt)`
(line 75-76). Server-side schema → Arrow type construction fails.
-
`sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala:524-525`
and `.../ArrowDeserializer.scala:437-439` have no cases for the new
`LocalDateTimeNanosEncoder` / `InstantNanosEncoder` — fall to `case _ => throw
new RuntimeException(s"Unsupported Encoder($encoder)/Vector($v)
combination.")`. Even if the Arrow schema mapping were added, the encoder ↔
vector dispatch would still fail.
Is the Connect / Arrow integration tracked in a follow-up ticket?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
