[
https://issues.apache.org/jira/browse/SPARK-59132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126350#comment-18126350
]
Dejan Krakovic commented on SPARK-59132:
----------------------------------------
h3. Implementation plan
The work is split into independently mergeable sub-tasks, following the four
efforts in the SPIP. Parquet storage is the only part that waits on
parquet-java (apache/parquet-java#3608, which needs a release and a version
bump in Spark).
||#||Sub-task||Component||Depends on||
|1|Catalyst type: FileType, FILE [EXTERNAL] DDL, JSON,
spark.sql.fileType.enabled (SPARK-60092)|SQL|-|
|2|Require EXTERNAL and reject non-comparable uses of FILE in table DDL|SQL|1|
|3|Physical representation: unsafe row, encoders, codegen, columnar,
Arrow|SQL|1|
|3b|File resolution layer: inline bytes, Hadoop FileSystem, 2GB bound|SQL|3|
|3c|FileType in PySpark and Spark Connect Python types|PySpark|1|
|4|Property access and casts (to/from BINARY, STRING, STRUCT, VARIANT)|SQL|3,
3b|
|5|TO_FILE, TRY_TO_FILE, CREATE_FILE, COPY_FILE, TRY_COPY_FILE, plus the
optimizer filter pushdown|SQL|3b, 4|
|6|FileRef and the UDF surface (Java and Python)|SQL|3b, 3c|
|7|Spark Connect support|Connect|3, 6|
|8|Parquet read/write, blocking other formats, cross-reader tests and
benchmarks|SQL|3, parquet-java release|
|9|Enable by default and SQL reference documentation|SQL|2 to 8|
After 1, PRs 2, 3 and 3c can proceed in parallel. spark.sql.fileType.enabled
stays off by default (on in tests) until 9, and must not be flipped before 2
lands, so that a bare FILE cannot be persisted in a table column.
Not planned here: FILE MANAGED, and Delta/Iceberg table-format support.
> FileType: A new data type to simplify handling of unstructured data in Spark
> ----------------------------------------------------------------------------
>
> Key: SPARK-59132
> URL: https://issues.apache.org/jira/browse/SPARK-59132
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Burak Yavuz
> Priority: Major
>
> We want to make it easy and cheap to work with files, such as PDFs, images,
> audio and video, from tables using Spark. Today you either copy the bytes of
> a file into a table as a binary column, or you keep the path to the file as a
> string and hope everything downstream knows what to do with it.
> We propose a new data type, *FileType* (spelled *FILE* in SQL), which holds
> the *path of a file* and a small amount of {*}metadata describing it{*}. The
> engine, the file format, and user defined functions all understand this type,
> so a query can filter, join and project files without reading their contents,
> and load the bytes only where it actually needs them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
