ranflarion opened a new issue, #27481:
URL: https://github.com/apache/datafusion/issues/27481
### Describe the bug
The Parquet opener hands `PhysicalExprAdapterFactory::create` the file's
Arrow schema as `physical_file_schema`, with the schema metadata arrow-rs
builds from the footer's key-value metadata. When the scan also reads a virtual
column (`TableSchema::virtual_columns`, for example `row_number`), the opener
first appends the virtual fields with `append_fields`, which builds its result
with `Schema::new(fields)` and drops the metadata. An adapter that reads the
file's metadata therefore works for every scan except the ones that ask for a
row number.
The same helper builds the logical file schema with virtual columns and the
decoder's stream schema, so those lose their metadata the same way.
### To Reproduce
On main at e2dc8b048, write a file with a key-value entry
```rust
let props = WriterProperties::builder()
.set_key_value_metadata(Some(vec![KeyValue::new(
"writer".to_string(),
"test".to_string(),
)]))
.build();
```
and scan it with an adapter factory that looks at the metadata it is given:
```rust
impl PhysicalExprAdapterFactory for RecordingFactory {
fn create(
&self,
logical_file_schema: SchemaRef,
physical_file_schema: SchemaRef,
) -> Result<Arc<dyn PhysicalExprAdapter>> {
println!("{:?}", physical_file_schema.metadata().get("writer"));
DefaultPhysicalExprAdapterFactory.create(logical_file_schema,
physical_file_schema)
}
}
```
A scan of the file's column prints `Some("test")`. With a `row_number`
virtual column added to the table schema
(`TableSchemaBuilder::with_virtual_columns`) and projected, it prints `None`.
### Expected behavior
`Some("test")` in both scans.
### Additional context
We read Spark's key-value entries (the writer version and its calendar
markers) in an adapter to decide per file how to convert dates and timestamps,
and scans that also read a row index lost them.
`append_fields` in `datafusion/datasource-parquet/src/opener/mod.rs` can
keep the metadata with `Schema::new_with_metadata(fields,
base.metadata().clone())`. On the path without virtual columns all three
schemas already carry it.
### Willingness to contribute
I plan to contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]