Max Gekk created SPARK-57463:
--------------------------------
Summary: Render nanosecond-precision timestamp types in the Thrift
server via the Types Framework
Key: SPARK-57463
URL: https://issues.apache.org/jira/browse/SPARK-57463
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
Umbrella: SPARK-56822 (Timestamps with nanosecond precision).
Implement the Types Framework thriftTypeName hook for the nanosecond-capable
timestamp types TIMESTAMP_NTZ(p) and TIMESTAMP_LTZ(p) (p in 7-9) so they are
usable over the Spark Thrift / JDBC server, reaching parity with the
microsecond TimestampType / TimestampNTZType.
Today TimestampNanosTypeApiOps does not override TypeApiOps.thriftTypeName (it
defaults to None). SparkExecuteStatementOperation resolves a column's Thrift
TTypeId via TypeApiOps(typ).flatMap(_.thriftTypeName), so nanosecond columns
have no Thrift type mapping. The reference TimeType ops returns
Some("STRING_TYPE") (TimeTypeApiOps.thriftTypeName).
Scope:
- Override thriftTypeName in TimestampNanosTypeApiOps to return
Some("STRING_TYPE") (rendered as a string at the column precision, consistent
with cast-to-string), or the most appropriate TTypeId.
- Verify SparkExecuteStatementOperation result-set rendering uses the framework
format/formatUTF8 path for the value.
- Tests under sql/hive-thriftserver covering NTZ and LTZ at precisions 7-9.
Note: related Hive CLI rendering through the framework is tracked by
SPARK-57386.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
