0lai0 commented on code in PR #5932:
URL: https://github.com/apache/datafusion-comet/pull/5932#discussion_r4034249279
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -911,6 +1083,51 @@ impl PhysicalExprAdapterFactory for
SparkPhysicalExprAdapterFactory {
None
};
+ // Compare the file's VARIANT annotations against the requested types
before handing the
+ // schemas to the default adapter. `adapted_physical_schema` is used
so that field-id and
+ // case-insensitive resolution has already aligned the two sides'
top-level names.
+ if !self.parquet_options.ignore_variant_annotation {
+ let mut physical_by_folded: HashMap<&str, usize> = HashMap::new();
+ for (i, name) in physical_folded.iter().enumerate() {
+ physical_by_folded.entry(name.as_str()).or_insert(i);
+ }
+ // When the read schema is known, check each requested root
against its requested
+ // type and skip roots Spark would not read. Requested roots share
their Spark names
+ // with the logical file schema, so the same fold pairs them.
+ let required_by_folded: Option<HashMap<String, &FieldRef>> =
+ self.required_schema.as_ref().map(|required| {
+ let mut map = HashMap::new();
+ for (field, folded) in required
+ .fields()
+ .iter()
+ .zip(fold_schema_names(required, case_sensitive))
Review Comment:
Done. Merged current main, and the requested-schema fold now propagates its
error through a fallible closure and transpose().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
