[
https://issues.apache.org/jira/browse/SPARK-42918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johan Lasperas updated SPARK-42918:
-----------------------------------
Description:
A first step towards allowing file format implementations to inject custom
metadata fields into plans is to make the handling of metadata attributes in
`FileSourceStrategy` more generic.
Today in `FileSourceStrategy` , the lists of constant and generated metadata
fields are created manually, checking for known generated fields on one hand
and considering the remaining fields as constant metadata fields. We need
instead to introduce a way of declaring metadata fields as generated or
constant directly in `FileFormat` and propagate that information to
`FileSourceStrategy`.
was:
A first step towards allowing file format implementations to inject custom
metadata columns into plans is to make the handling of metadata attributes in
`FileSourceStrategy` more generic.
Today in `FileSourceStrategy` , the lists of constant and generated metadata
columns are created manually, checking for known generated columns on one hand
and considering the remaining columns as constant metadata columns. We need
instead to introduce a way of declaring metadata columns as generated or
constant directly in `FileFormat` and propagate that information to
`FileSourceStrategy`.
> Generalize handling of metadata attributes in FileSourceStrategy
> ----------------------------------------------------------------
>
> Key: SPARK-42918
> URL: https://issues.apache.org/jira/browse/SPARK-42918
> Project: Spark
> Issue Type: Improvement
> Components: Optimizer
> Affects Versions: 3.4.1
> Reporter: Johan Lasperas
> Priority: Minor
>
> A first step towards allowing file format implementations to inject custom
> metadata fields into plans is to make the handling of metadata attributes in
> `FileSourceStrategy` more generic.
> Today in `FileSourceStrategy` , the lists of constant and generated metadata
> fields are created manually, checking for known generated fields on one hand
> and considering the remaining fields as constant metadata fields. We need
> instead to introduce a way of declaring metadata fields as generated or
> constant directly in `FileFormat` and propagate that information to
> `FileSourceStrategy`.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
