peterxcli opened a new issue, #25251: URL: https://github.com/apache/datafusion/issues/25251
### Is your feature request related to a problem or challenge? Comet's projected Variant scans need to infer each file's Arrow schema from its physical Parquet schema, ignore advisory `ARROW:schema`, and map ENUM leaves to Utf8 while retaining raw BINARY. The shredded layout can differ between files and is only available after loading that file's footer. #22360 already supports an explicit `PartitionedFile.arrow_schema`. That solves this when the complete physical file schema is known before opening. Arrow's `ArrowReaderOptions::with_schema` also supports nested Binary-to-Utf8 hints, so a new Arrow ENUM conversion API is unnecessary for this case. The remaining gap is configuring that schema during lazy file opening. On DataFusion `9082d6b10c29b72d56bede3d8e353d9d61fde542`, [the opener](https://github.com/apache/datafusion/blob/9082d6b10c29b72d56bede3d8e353d9d61fde542/datafusion/datasource-parquet/src/opener/mod.rs#L922-L1022) creates options from the already populated `PartitionedFile`, then calls `ArrowReaderMetadata::load_async`. `ParquetFileReaderFactory` returns an `AsyncFileReader`, whose `get_metadata` receives borrowed options and returns `ParquetMetaData`; it cannot return a newly derived Arrow schema or revised reader options to the opener. Comet currently rewrites the returned footer's Arrow schema hint. Rebuilding public `FileMetaData` cannot retain its private decryptor state, so this path requires an encrypted Variant fallback. ### Describe the solution you'd like Allow a ParquetSource reader policy to derive a supplied Arrow schema from the loaded physical `SchemaDescriptor` during file opening, before advisory Arrow schema parsing and filter/schema preparation. Reuse Arrow's existing `with_schema` support and the original `Arc<ParquetMetaData>`, preserving decryption properties, page indexes and existing metadata-cache behavior. The policy must be optional per source, leave ordinary scans unchanged, and support files with different nested schemas without fetching their footers during planning. A specific trait or callback design is not prescribed here. ### Describe alternatives you've considered - Populate `PartitionedFile.arrow_schema` through #22360 when the full per-file schema is already available. For Comet's current path, this requires moving footer loading ahead of lazy opening. - Rewrite the footer hint: works for unencrypted files but rebuilds metadata and couples schema policy to storage metadata. - `with_skip_arrow_metadata(true)` handles hint suppression alone; the ENUM policy still needs the physical annotation. - #25192 addresses recursion in existing table-schema coercions. The table's logical Variant schema does not contain the per-file shredded leaves or distinguish physical ENUM from BINARY, so recursion alone does not solve this case. ### Additional context Tracked in apache/datafusion-comet#5477, implemented as a workaround in apache/datafusion-comet#5868. This requests a reader integration point, not a change to DataFusion's default ENUM interpretation or an expansion of the canonical Variant format. A local check against arrow-rs `4cd8be954f6bc6b6dd265140207365b59a9900ec` passed a supplied schema for `nested { e: ENUM, b: BINARY }`: `e` became Utf8, `b` stayed Binary, an invalid advisory `ARROW:schema` was ignored, and `Arc::ptr_eq(reader_metadata.metadata(), original_metadata)` remained true. No encrypted scan was run; preserving decryption behavior needs coverage when implementing this integration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
