ranflarion opened a new issue, #27481:
URL: https://github.com/apache/datafusion/issues/27481

   ### Describe the bug
   
   The Parquet opener hands `PhysicalExprAdapterFactory::create` the file's 
Arrow schema as `physical_file_schema`, with the schema metadata arrow-rs 
builds from the footer's key-value metadata. When the scan also reads a virtual 
column (`TableSchema::virtual_columns`, for example `row_number`), the opener 
first appends the virtual fields with `append_fields`, which builds its result 
with `Schema::new(fields)` and drops the metadata. An adapter that reads the 
file's metadata therefore works for every scan except the ones that ask for a 
row number.
   
   The same helper builds the logical file schema with virtual columns and the 
decoder's stream schema, so those lose their metadata the same way.
   
   ### To Reproduce
   
   On main at e2dc8b048, write a file with a key-value entry
   
   ```rust
   let props = WriterProperties::builder()
       .set_key_value_metadata(Some(vec![KeyValue::new(
           "writer".to_string(),
           "test".to_string(),
       )]))
       .build();
   ```
   
   and scan it with an adapter factory that looks at the metadata it is given:
   
   ```rust
   impl PhysicalExprAdapterFactory for RecordingFactory {
       fn create(
           &self,
           logical_file_schema: SchemaRef,
           physical_file_schema: SchemaRef,
       ) -> Result<Arc<dyn PhysicalExprAdapter>> {
           println!("{:?}", physical_file_schema.metadata().get("writer"));
           DefaultPhysicalExprAdapterFactory.create(logical_file_schema, 
physical_file_schema)
       }
   }
   ```
   
   A scan of the file's column prints `Some("test")`. With a `row_number` 
virtual column added to the table schema 
(`TableSchemaBuilder::with_virtual_columns`) and projected, it prints `None`.
   
   ### Expected behavior
   
   `Some("test")` in both scans.
   
   ### Additional context
   
   We read Spark's key-value entries (the writer version and its calendar 
markers) in an adapter to decide per file how to convert dates and timestamps, 
and scans that also read a row index lost them.
   
   `append_fields` in `datafusion/datasource-parquet/src/opener/mod.rs` can 
keep the metadata with `Schema::new_with_metadata(fields, 
base.metadata().clone())`. On the path without virtual columns all three 
schemas already carry it.
   
   ### Willingness to contribute
   
   I plan to contribute a fix for this bug independently
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to