sam-1112 commented on code in PR #7196:
URL: https://github.com/apache/datafusion-comet/pull/7196#discussion_r4369358873
##########
native/core/src/parquet/field_resolver.rs:
##########
@@ -204,6 +241,55 @@ impl<'a> FieldResolver<'a> {
)),
}
}
+
+ /// Resolve claims in file order, with id matches ahead of shadows and
name matches.
+ /// Repeated requested ids keep the last claimant, as Spark parquet-mr
does. Duplicate file ids are
+ /// checked across the whole requested group, including unprojected root
fields (#7006).
+ /// Name ambiguity is left to the caller: the root checks referenced
columns only, whereas
+ /// nested conversion validates every requested field in
[`Self::resolve_all`].
+ pub(crate) fn file_claims(&self) -> DataFusionResult<Vec<FieldClaim>> {
+ let mut claims = vec![FieldClaim::Unchanged; self.file.len()];
+ for requested in 0..self.requested.len() {
+ match self.select(requested)? {
+ Selection::ById(matched) => {
+ if let Some(file) = matched {
+ claims[file].claim(FieldClaim::IdMatch(requested));
+ }
+ for &file in self.name_matches(requested) {
+ claims[file].claim(FieldClaim::Shadowed);
+ }
+ }
+ Selection::ByName(matches) => {
+ for &file in matches {
+ claims[file].claim(FieldClaim::NameMatch(requested));
+ }
+ }
+ }
+ }
+ Ok(claims)
+ }
+
+ /// Resolve a nested group in requested order, giving each file field to
its strongest
+ /// claimant only. Missing and losing fields read NULL. Validate
independent selections
+ /// first, in requested order, so arbitration cannot hide ambiguity or
change which of
+ /// several invalid selections reports its error first.
+ pub(crate) fn resolve_all(&self) -> DataFusionResult<Vec<Option<usize>>> {
+ let mut resolved = (0..self.requested.len())
+ .map(|requested| self.resolve(requested))
+ .collect::<DataFusionResult<Vec<_>>>()?;
+ let claims = self.file_claims()?;
Review Comment:
`resolve_all()` now selects each child once and records claims from that
validated selection, preserving requested-order ambiguity errors.
I added `parquet_field_resolution.rs` to benchmark conversion and
decode-plus-conversion with 64 unique-ID and renamed children at 64/8192 rows
per batch. Nine alternating paired runs showed ~9.2% and ~4.2% lower elapsed
time respectively at 64 rows/batch; the 8192-row combined result was
essentially flat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
