[
https://issues.apache.org/jira/browse/SPARK-59132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141577#comment-18141577
]
Dejan Krakovic edited comment on SPARK-59132 at 11/13/26, 11:42 PM:
--------------------------------------------------------------------
h3. Implementation plan
Parquet storage does not wait on a parquet-java release: FILE is first stored
as a group of the FILE fields without the logical type annotation, and the
annotation follows once a parquet-java release includes
[apache/parquet-java#3608|https://github.com/apache/parquet-java/pull/3608]. A
parquet-java version without FILE support already reads annotated files as
plain groups (apache/parquet-java#3669).
Status:
* SPARK-60092 (done): the FileType Catalyst type, the FILE / FILE EXTERNAL DDL
and the JSON schema format.
* SPARK-60386 (done): FILE EXTERNAL in table columns, plain FILE in routine
signatures, and rejecting FILE where it would need equality or an ordering.
Remaining work, one JIRA and PR each. Sub-tasks are filed when their PR is
ready, so the split below may still change:
||#||Slice||Component||Depends on||
|1a|Physical representation: physical type, UnsafeRow and codegen access,
InternalRow / Row conversion, column vectors|SQL|SPARK-60386|
|1b|FileRef value class (data only) and encoders|SQL|1a|
|2|Field access (f.uri, f.offset, f.size, f.content_type, f.checksum; inline
stays internal) and casts from STRUCT and VARIANT to FILE|SQL|1a|
|3|Parquet read and write of FILE as a group of the FILE fields, a clear error
for formats that cannot store FILE, and benchmarks|SQL|1a|
|3b|Parquet FILE logical type annotation on write, recognizing annotated groups
on read, validating values on read, and cross-reader tests|SQL|3, and a
parquet-java release with #3608|
|4a|Byte access: resolving the bytes of a value (inline, or uri with offset /
size, through the Hadoop FileSystem), and casts to BINARY and STRING bounded at
2GB|SQL|1b, 3|
|4b|TO_FILE and TRY_TO_FILE, and pushing filters below TO_FILE|SQL|4a|
|4c|CREATE_FILE, COPY_FILE and TRY_COPY_FILE|SQL|4a|
|5|FileRef I/O API (open, asLocalFile, fromBytes, fromLocalFile) and JVM UDFs
over FILE|SQL|4a|
|6a|PySpark: FileType in pyspark.sql.types (type and schema
only)|PySpark|SPARK-60092|
|6b|PySpark: values and Arrow conversion, Python FileRef, and Python UDFs /
UDTFs|PySpark|1b, 5|
|7a|Spark Connect: FileType in the DataType proto and schema conversion in the
Scala and Python clients|Connect|SPARK-60092|
|7b|Spark Connect: value conversion, Arrow and result serialization|Connect|1b|
|8|Enable spark.sql.fileType.enabled by default, and SQL reference
documentation|SQL|all of the above, SPARK-61297|
Can run in parallel:
* Now: 6a and 7a (type level only), and the follow-ups below.
* After 1a: 1b, 2 and 3.
* After 4a: 4b, 4c and 5. 7b can run alongside 4 once 1b is in.
* 3b waits only on the parquet-java release. Parquet files written before it
carry no FILE annotation, so it should land before 8.
spark.sql.fileType.enabled stays off by default (on in tests) until 8.
Follow-ups from the SPARK-60386 review:
* SPARK-61297: FILE checks for Declarative Pipelines tables and for writes to
tables that accept any schema. Must be fixed before 8.
* SPARK-61298: one rule for FILE in the output of row-returning Python
functions (with 6b).
* SPARK-61299: declare equality and ordering support per data type, and share
the checks for MAP, VARIANT and FILE.
* SPARK-61300: consistent set operation errors in both analyzers.
* SPARK-61301: FILE error messages, SQLSTATEs and helpers.
* SPARK-61302: whether FileType should define equality.
* SPARK-61303: the FILE return type of all V2 catalog functions.
* SPARK-61304: whether to reject FILE in aggregates that deduplicate or hash
their input (with 4a).
Not planned here: FILE MANAGED, and Delta / Iceberg table format support.
was (Author: JIRAUSER307461):
h3. Implementation plan
Parquet storage does not wait on a parquet-java release: FILE is first stored
as a group of the FILE fields without the logical type annotation, and the
annotation follows once a parquet-java release includes
[apache/parquet-java#3608|https://github.com/apache/parquet-java/pull/3608]. A
parquet-java version without FILE support already reads annotated files as
plain groups (apache/parquet-java#3669).
Status:
* SPARK-60092 (done): the FileType Catalyst type, the FILE / FILE EXTERNAL DDL
and the JSON schema format.
* SPARK-60386 (in review, [PR
#59640|https://github.com/apache/spark/pull/59640]): FILE EXTERNAL in table
columns, plain FILE in routine signatures, and rejecting FILE where it would
need equality or an ordering.
Remaining work, one JIRA and PR each. Sub-tasks are filed when their PR is
ready, so the split below may still change:
||#||Slice||Component||Depends on||
|1a|Physical representation: physical type, UnsafeRow and codegen access,
InternalRow / Row conversion, column vectors|SQL|SPARK-60386|
|1b|FileRef value class (data only) and encoders|SQL|1a|
|2|Field access (f.uri, f.offset, f.size, f.content_type, f.checksum; inline
stays internal) and casts from STRUCT and VARIANT to FILE|SQL|1a|
|3|Parquet read and write of FILE as a group of the FILE fields, a clear error
for formats that cannot store FILE, and benchmarks|SQL|1a|
|3b|Parquet FILE logical type annotation on write, recognizing annotated groups
on read, validating values on read, and cross-reader tests|SQL|3, and a
parquet-java release with #3608|
|4a|Byte access: resolving the bytes of a value (inline, or uri with offset /
size, through the Hadoop FileSystem), and casts to BINARY and STRING bounded at
2GB|SQL|1b, 3|
|4b|TO_FILE and TRY_TO_FILE, and pushing filters below TO_FILE|SQL|4a|
|4c|CREATE_FILE, COPY_FILE and TRY_COPY_FILE|SQL|4a|
|5|FileRef I/O API (open, asLocalFile, fromBytes, fromLocalFile) and JVM UDFs
over FILE|SQL|4a|
|6a|PySpark: FileType in pyspark.sql.types (type and schema
only)|PySpark|SPARK-60092|
|6b|PySpark: values and Arrow conversion, Python FileRef, and Python UDFs /
UDTFs|PySpark|1b, 5|
|7a|Spark Connect: FileType in the DataType proto and schema conversion in the
Scala and Python clients|Connect|SPARK-60092|
|7b|Spark Connect: value conversion, Arrow and result serialization|Connect|1b|
|8|Enable spark.sql.fileType.enabled by default, and SQL reference
documentation|SQL|all of the above, SPARK-61297|
Can run in parallel:
* Now: 6a and 7a (type level only), and the follow-ups below.
* After 1a: 1b, 2 and 3.
* After 4a: 4b, 4c and 5. 7b can run alongside 4 once 1b is in.
* 3b waits only on the parquet-java release. Parquet files written before it
carry no FILE annotation, so it should land before 8.
spark.sql.fileType.enabled stays off by default (on in tests) until 8.
Follow-ups from the SPARK-60386 review:
* SPARK-61297: FILE checks for Declarative Pipelines tables and for writes to
tables that accept any schema. Must be fixed before 8.
* SPARK-61298: one rule for FILE in the output of row-returning Python
functions (with 6b).
* SPARK-61299: declare equality and ordering support per data type, and share
the checks for MAP, VARIANT and FILE.
* SPARK-61300: consistent set operation errors in both analyzers.
* SPARK-61301: FILE error messages, SQLSTATEs and helpers.
* SPARK-61302: whether FileType should define equality.
* SPARK-61303: the FILE return type of all V2 catalog functions.
* SPARK-61304: whether to reject FILE in aggregates that deduplicate or hash
their input (with 4a).
Not planned here: FILE MANAGED, and Delta / Iceberg table format support.
> FileType: A new data type to simplify handling of unstructured data in Spark
> ----------------------------------------------------------------------------
>
> Key: SPARK-59132
> URL: https://issues.apache.org/jira/browse/SPARK-59132
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Burak Yavuz
> Priority: Major
>
> We want to make it easy and cheap to work with files, such as PDFs, images,
> audio and video, from tables using Spark. Today you either copy the bytes of
> a file into a table as a binary column, or you keep the path to the file as a
> string and hope everything downstream knows what to do with it.
> We propose a new data type, *FileType* (spelled *FILE* in SQL), which holds
> the *path of a file* and a small amount of {*}metadata describing it{*}. The
> engine, the file format, and user defined functions all understand this type,
> so a query can filter, join and project files without reading their contents,
> and load the bytes only where it actually needs them.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
