[
https://issues.apache.org/jira/browse/SPARK-62112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62112:
-----------------------------------
Labels: pull-request-available (was: )
> Add the physical representation and the FileRef value of the FILE type
> ----------------------------------------------------------------------
>
> Key: SPARK-62112
> URL: https://issues.apache.org/jira/browse/SPARK-62112
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Dejan Krakovic
> Priority: Major
> Labels: pull-request-available
>
> Slice 1 of the SPARK-59132 implementation plan: the physical representation
> and the value of FILE.
> - Register FileType in the Types Framework (SPARK-53504). It is stored as a
> struct of FileType.FIELDS (uri, offset, size, content_type, checksum,
> inline), so with the generic row access of SPARK-56076 it works through the
> row readers and writers, projections and code generation without per-type
> changes there.
> - Column vectors for a struct-backed framework type: WritableColumnVector,
> ConstantColumnVector and ColumnVectorUtils build one child vector per field,
> which SPARK-56076 left to the first type that needs it.
> - FileRef, the user-facing Java value class of FILE: the nullable getters
> uri, offset, size, contentType and checksum, with equals, hashCode and
> toString. inline stays internal. The I/O methods (open, asLocalFile,
> fromBytes, fromLocalFile) come with slice 5.
> - The FileRef encoder, the mapping of FileRef in ScalaReflection and
> JavaTypeInference, the conversions between FileRef and InternalRow, and
> Row.getFile.
> spark.sql.fileType.enabled stays off by default. Field access, casts, Parquet
> storage, JSON output, and Python and Spark Connect values are separate slices.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]