Yuchuan Huang created SPARK-61059:
-------------------------------------
Summary: Support derived file source metadata fields computed from
other metadata fields
Key: SPARK-61059
URL: https://issues.apache.org/jira/browse/SPARK-61059
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yuchuan Huang
A FileFormat can currently declare two kinds of `_metadata` struct fields:
- constant fields (FileSourceConstantMetadataStructField), whose values are
constant per file and are appended to the scan output;
- generated fields (FileSourceGeneratedMetadataStructField), whose values are
produced per row by the file reader through an internal column.
There is no way to declare a metadata field whose value is computed from other
metadata fields, e.g. `row_id = coalesce(materialized_row_id, base_row_id +
row_index)` as used by row tracking in table formats. Today such formats have
to inject custom analyzer/optimizer rules to compute these values.
This ticket proposes a third kind, FileSourceDerivedMetadataStructField: a
metadata field that carries a deterministic scalar Catalyst expression over its
constant or generated siblings (referenced by name). FileSourceStrategy reads
the required inputs (even if they are not selected) and evaluates the
expression in the scan's metadata projection, so analyzed and optimized plans
keep the ordinary `_metadata` shape.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
