Dejan Krakovic created SPARK-62112:
--------------------------------------
Summary: Add the physical representation and the FileRef value of
the FILE type
Key: SPARK-62112
URL: https://issues.apache.org/jira/browse/SPARK-62112
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Dejan Krakovic
Slice 1 of the SPARK-59132 implementation plan: the physical representation and
the value of FILE.
- Register FileType in the Types Framework (SPARK-53504). It is stored as a
struct of FileType.FIELDS (uri, offset, size, content_type, checksum, inline),
so with the generic row access of SPARK-56076 it works through the row readers
and writers, projections and code generation without per-type changes there.
- Column vectors for a struct-backed framework type: WritableColumnVector,
ConstantColumnVector and ColumnVectorUtils build one child vector per field,
which SPARK-56076 left to the first type that needs it.
- FileRef, the user-facing Java value class of FILE: the nullable getters uri,
offset, size, contentType and checksum, with equals, hashCode and toString.
inline stays internal. The I/O methods (open, asLocalFile, fromBytes,
fromLocalFile) come with slice 5.
- The FileRef encoder, the mapping of FileRef in ScalaReflection and
JavaTypeInference, the conversions between FileRef and InternalRow, and
Row.getFile.
spark.sql.fileType.enabled stays off by default. Field access, casts, Parquet
storage, JSON output, and Python and Spark Connect values are separate slices.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]