[ 
https://issues.apache.org/jira/browse/SPARK-62112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62112:
-----------------------------------
    Labels: pull-request-available  (was: )

> Add the physical representation and the FileRef value of the FILE type
> ----------------------------------------------------------------------
>
>                 Key: SPARK-62112
>                 URL: https://issues.apache.org/jira/browse/SPARK-62112
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Dejan Krakovic
>            Priority: Major
>              Labels: pull-request-available
>
> Slice 1 of the SPARK-59132 implementation plan: the physical representation 
> and the value of FILE.
> - Register FileType in the Types Framework (SPARK-53504). It is stored as a 
> struct of FileType.FIELDS (uri, offset, size, content_type, checksum, 
> inline), so with the generic row access of SPARK-56076 it works through the 
> row readers and writers, projections and code generation without per-type 
> changes there.
> - Column vectors for a struct-backed framework type: WritableColumnVector, 
> ConstantColumnVector and ColumnVectorUtils build one child vector per field, 
> which SPARK-56076 left to the first type that needs it.
> - FileRef, the user-facing Java value class of FILE: the nullable getters 
> uri, offset, size, contentType and checksum, with equals, hashCode and 
> toString. inline stays internal. The I/O methods (open, asLocalFile, 
> fromBytes, fromLocalFile) come with slice 5.
> - The FileRef encoder, the mapping of FileRef in ScalaReflection and 
> JavaTypeInference, the conversions between FileRef and InternalRow, and 
> Row.getFile.
> spark.sql.fileType.enabled stays off by default. Field access, casts, Parquet 
> storage, JSON output, and Python and Spark Connect values are separate slices.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to