Gengliang Wang created SPARK-61909:
--------------------------------------

             Summary: Support more data types in native data sources, such as 
intervals, time and variant
                 Key: SPARK-61909
                 URL: https://issues.apache.org/jira/browse/SPARK-61909
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


Version 1 of the native data source interface ({{spark_datasource.h}}) supports 
boolean, the integer types, float, double, decimal, string, binary, date, 
timestamp, timestamp_ntz, array, map and struct. Spark fails an operation whose 
schema has another type, such as an interval, variant or time, before it passes 
the schema to the library. The header says that a later version may support 
other types, for the data sources that declare a capability for them.

Proposal: define the Arrow types of more Spark types, such as year-month and 
day-time intervals, time and variant, and a capability with which a data source 
declares that it supports them, and map them in the Rust SDK (SPARK-61282).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to