[
https://issues.apache.org/jira/browse/SPARK-42918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johan Lasperas updated SPARK-42918:
-----------------------------------
Description:
A first step towards allowing file format implementations to inject custom
metadata columns into plans is to make the handling of metadata attributes in
`FileSourceStrategy` more generic.
Today in `FileSourceStrategy` , the lists of constant and generated metadata
columns are created manually, checking for known generated columns on one hand
and considering the remaining columns as constant metadata columns. We need
instead to introduce a way of declaring metadata columns as generated or
constant directly in `FileFormat` and propagate that information to
`FileSourceStrategy`.
was:A first step towards allowing file format implementations to inject
custom metadata columns into plans is to make handling of metadata attributes
in `FileSourceStrategy` more generic.
> Generalize handling of metadata attributes in FileSourceStrategy
> ----------------------------------------------------------------
>
> Key: SPARK-42918
> URL: https://issues.apache.org/jira/browse/SPARK-42918
> Project: Spark
> Issue Type: Improvement
> Components: Optimizer
> Affects Versions: 3.4.1
> Reporter: Johan Lasperas
> Priority: Minor
>
> A first step towards allowing file format implementations to inject custom
> metadata columns into plans is to make the handling of metadata attributes in
> `FileSourceStrategy` more generic.
> Today in `FileSourceStrategy` , the lists of constant and generated metadata
> columns are created manually, checking for known generated columns on one
> hand and considering the remaining columns as constant metadata columns. We
> need instead to introduce a way of declaring metadata columns as generated or
> constant directly in `FileFormat` and propagate that information to
> `FileSourceStrategy`.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
