Burak Yavuz created SPARK-59132:
-----------------------------------
Summary: FileType: A new data type to simplify handling of
unstructured data in Spark
Key: SPARK-59132
URL: https://issues.apache.org/jira/browse/SPARK-59132
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: Burak Yavuz
We want to make it easy and cheap to work with files, such as PDFs, images,
audio and video, from tables using Spark. Today you either copy the bytes of a
file into a table as a binary column, or you keep the path to the file as a
string and hope everything downstream knows what to do with it.
We propose a new data type, *FileType* (spelled *FILE* in SQL), which holds the
*path of a file* and a small amount of {*}metadata describing it{*}. The
engine, the file format, and user defined functions all understand this type,
so a query can filter, join and project files without reading their contents,
and load the bytes only where it actually needs them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
