Dejan Krakovic created SPARK-62112:
--------------------------------------

             Summary: Add the physical representation and the FileRef value of 
the FILE type
                 Key: SPARK-62112
                 URL: https://issues.apache.org/jira/browse/SPARK-62112
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Dejan Krakovic


Slice 1 of the SPARK-59132 implementation plan: the physical representation and 
the value of FILE.

- Register FileType in the Types Framework (SPARK-53504). It is stored as a 
struct of FileType.FIELDS (uri, offset, size, content_type, checksum, inline), 
so with the generic row access of SPARK-56076 it works through the row readers 
and writers, projections and code generation without per-type changes there.
- Column vectors for a struct-backed framework type: WritableColumnVector, 
ConstantColumnVector and ColumnVectorUtils build one child vector per field, 
which SPARK-56076 left to the first type that needs it.
- FileRef, the user-facing Java value class of FILE: the nullable getters uri, 
offset, size, contentType and checksum, with equals, hashCode and toString. 
inline stays internal. The I/O methods (open, asLocalFile, fromBytes, 
fromLocalFile) come with slice 5.
- The FileRef encoder, the mapping of FileRef in ScalaReflection and 
JavaTypeInference, the conversions between FileRef and InternalRow, and 
Row.getFile.

spark.sql.fileType.enabled stays off by default. Field access, casts, Parquet 
storage, JSON output, and Python and Spark Connect values are separate slices.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to