sam-1112 commented on code in PR #7196:
URL: https://github.com/apache/datafusion-comet/pull/7196#discussion_r4369351920
##########
native/core/src/parquet/field_resolver.rs:
##########
@@ -47,6 +47,43 @@ pub(crate) enum Selection<'a> {
ByName(&'a [usize]),
}
+/// The strongest claim on one file field. Root remapping and nested
conversion give a file
+/// field to this claimant only. Repeated id claims keep the last requested
field, matching
+/// Spark parquet-mr's `ParquetRowConverter` id-to-index `toMap`. Equal name
claims keep the
+/// earliest requested field, retaining the root remap's name-matching
behavior.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum FieldClaim {
+ /// The requested field selects this file field by Parquet field id.
+ IdMatch(usize),
+ /// An id-bearing requested field has this file field's folded name. It
must not reach this
+ /// field by name when its id is missing or selects a different field.
Only an id match can
+ /// override this claim. The root hides it with a placeholder; nested
fields read NULL.
+ Shadowed,
+ /// The requested field selects this file field by folded name.
+ NameMatch(usize),
+ /// No requested field claims this file field. The root keeps its original
name.
+ Unchanged,
+}
+
+impl FieldClaim {
+ fn strength(self) -> u8 {
+ match self {
+ Self::IdMatch(_) => 3,
+ Self::Shadowed => 2,
+ Self::NameMatch(_) => 1,
+ Self::Unchanged => 0,
+ }
+ }
+
+ fn claim(&mut self, claim: Self) {
+ if claim.strength() > self.strength()
+ || matches!((*self, claim), (Self::IdMatch(_), Self::IdMatch(_)))
Review Comment:
Thanks!! I have fixed by checking required roots against the actual data
schema produced by `nativeDataSchema()`. With field-ID reads enabled, the
native scan now falls back when a required root shares its ID with another
serialized data root. Added regressions that project `x` and `y` separately in
both schema orders, comparing against both Spark readers and asserting the
fallback reason. Both aliases return `7`. Control cases confirm that selecting
an unrelated unique-ID root or disabling field-ID matching still uses the
native scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
