ranflarion opened a new pull request, #27486:
URL: https://github.com/apache/datafusion/pull/27486

   ## Which issue does this PR close?
   
   - Closes #27481.
   
   ## Rationale for this change
   
   A `PhysicalExprAdapterFactory` receives the Parquet file's schema metadata 
(the footer's key-value metadata) in `physical_file_schema`, except when the 
scan also reads a virtual column such as `row_number`, where it receives none. 
An adapter that reads that metadata to decide how to adapt a file stops working 
as soon as a row number is requested.
   
   ## What changes are included in this PR?
   
   `append_fields`, which adds the virtual fields to a schema, keeps the base 
schema's metadata instead of building its result with `Schema::new`. It has 
three callers: the physical file schema handed to the adapter factory and the 
simplifier, the logical file schema with virtual columns, and the decoder's 
stream schema. Each now carries the metadata it already has on the path without 
virtual columns (`build_projection_read_plan` builds the projected schema with 
the file schema's metadata).
   
   ## What is the testing strategy for this PR?
   
   `test_row_index_keeps_file_schema_metadata_for_expr_adapter` writes a file 
with a key-value entry and opens it through an adapter factory that records the 
entry it sees, with and without a `row_number` column. On main the read with 
the virtual column records `None`.
   
   ## Are there any user-facing changes?
   
   No API changes. The schemas the opener derives for a scan with virtual 
columns keep their metadata.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to