0lai0 commented on code in PR #5932:
URL: https://github.com/apache/datafusion-comet/pull/5932#discussion_r4029395323
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -911,6 +1088,29 @@ impl PhysicalExprAdapterFactory for
SparkPhysicalExprAdapterFactory {
None
};
+ // Compare the file's VARIANT annotations against the requested types
before handing the
+ // schemas to the default adapter. `adapted_physical_schema` is used
so that field-id and
+ // case-insensitive resolution has already aligned the two sides'
top-level names.
+ if !self.parquet_options.ignore_variant_annotation {
+ let mut physical_by_folded: HashMap<&str, usize> = HashMap::new();
+ for (i, name) in physical_folded.iter().enumerate() {
+ physical_by_folded.entry(name.as_str()).or_insert(i);
+ }
+ let mut path = Vec::new();
+ for (logical_field, folded) in
logical_file_schema.fields().iter().zip(&logical_folded)
Review Comment:
Thanks @sunchao. Sure, fixed. The validation now uses the requested read
schema. `init_datasource_exec `passes required_schema to the factory via
`with_required_schema`, so unrequested roots are skipped, and requested fields
are still checked eagerly in `create()`, including empty files. Regressions in
`parquet_exec/variant_tests.rs` cover an omitted annotated root (reads fine)
and a requested one (rejected).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
