Gengliang Wang created SPARK-61909:
--------------------------------------
Summary: Support more data types in native data sources, such as
intervals, time and variant
Key: SPARK-61909
URL: https://issues.apache.org/jira/browse/SPARK-61909
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
Version 1 of the native data source interface ({{spark_datasource.h}}) supports
boolean, the integer types, float, double, decimal, string, binary, date,
timestamp, timestamp_ntz, array, map and struct. Spark fails an operation whose
schema has another type, such as an interval, variant or time, before it passes
the schema to the library. The header says that a later version may support
other types, for the data sources that declare a capability for them.
Proposal: define the Arrow types of more Spark types, such as year-month and
day-time intervals, time and variant, and a capability with which a data source
declares that it supports them, and map them in the Rust SDK (SPARK-61282).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]