cblp opened a new pull request, #26856:
URL: https://github.com/apache/datafusion/pull/26856
## Which issue does this PR close?
- Closes #26855.
## Rationale for this change
Expressions pushed into a Parquet scan, such as `arrow_metadata(col)` or
extension-aware UDFs, can see a column without its field metadata: `SELECT
arrow_metadata(u) FROM t` returns `{}` although the table schema has
`ARROW:extension:name` on `u`. See #26855 for a reproducer.
The Parquet opener replaces column references with literals for partition
columns and for columns that the file statistics prove constant (`min == max`).
The literals had no field metadata, so expressions evaluated on them lost the
extension type. This happens for example for every column of a one-row file.
## What changes are included in this PR?
- New function `replace_columns_with_literals_keeping_metadata` in
`datafusion-physical-expr-adapter`, next to `replace_columns_with_literals`. It
takes the metadata of each literal from the field with the same name in a given
schema. `DefaultPhysicalExprAdapter` already creates its literals for missing
columns the same way.
- The Parquet opener uses it with the table schema, for the projection and
for the predicate.
`replace_columns_with_literals` is unchanged.
## What is the testing strategy for this PR?
- `pushed_down_projection_sees_field_metadata` in
`datafusion/core/tests/parquet/schema.rs`: `arrow_metadata(id,
'ARROW:extension:name')` over a one-row Parquet file.
- A new case in `cast_extension_type_metadata.slt`: a `UUID` column written
with `COPY ... TO` and read back. It sets `skip_metadata = false` because until
#26854 is merged, schema inference drops extension types by default (#26851).
- A unit test for `replace_columns_with_literals_keeping_metadata`.
Both integration tests fail without the change in the opener.
## Are there any user-facing changes?
Expressions evaluated in a Parquet scan now see the field metadata of the
table schema for partition columns and for columns that the statistics prove
constant.
There is one new public function,
`replace_columns_with_literals_keeping_metadata`, and no breaking API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
