[
https://issues.apache.org/jira/browse/SPARK-60092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60092:
-----------------------------------
Labels: pull-request-available (was: )
> Add FileType Catalyst data type with FILE EXTERNAL DDL and JSON schema support
> ------------------------------------------------------------------------------
>
> Key: SPARK-60092
> URL: https://issues.apache.org/jira/browse/SPARK-60092
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
> Affects Versions: 4.3.0
> Reporter: Dejan Krakovic
> Priority: Major
> Labels: pull-request-available
>
> Part 1 of the FILE type SPIP (SPARK-59132).
>
> Add the FileType Catalyst data type, independent of parquet-java, so that
> downstream consumers such as delta-spark, which represent column types as
> Spark Catalyst DataTypes, have a type to refer to. There is currently no
> FileType in OSS Spark.
>
> Scope:
> - FileType, a DataType that is physically a struct but does not extend
> StructType, with SQL name "FILE EXTERNAL" and JSON name "file".
> - The FILE keyword (non-reserved) and the FILE EXTERNAL DDL syntax. EXTERNAL
> is required when declaring a FILE column.
> - Schema parsing and serialization: DataType.fromJson / json and
> DataType.fromDDL / sql round trip, including FILE nested in array, map
> value and struct types.
> - DataTypes.FileType for the Java API, and keyword documentation in
> sql-ref-ansi-compliance.md.
>
> Out of scope (later parts): physical representation (unsafe row, codegen,
> encoders), casts, TO_FILE and related functions, comparability checks, the
> Parquet schema converters, the UDF surface (FileRef) and Spark Connect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
