sunchao commented on PR #5654: URL: https://github.com/apache/datafusion-comet/pull/5654#issuecomment-5682335533
Reviewed head `1fa518b54` against base `1d0ce5fe8`. **One P2 regression remains.** ### P2: Footer validation rejects supported duplicate-root reads [parquet_support.rs:352](https://github.com/apache/datafusion-comet/blob/1fa518b54ea68de77577dc61be74a470e181c51b/native/core/src/parquet/parquet_support.rs#L352) treats the root schema as a nested struct, applying the new duplicate-name rejection before the adapter can select the first root column. **Trigger:** a file contains `d`, `d`, and `a(id=7)`. Request `d` by name and `a` by ID, with case sensitivity and field-ID reads enabled. | Reader | Result | |---|---| | Spark 4.1.3 vectorized | `(1,100), (2,200), (3,300)` | | Base native scan harness | Same rows | | PR native scan harness | Duplicate-field error for `d` | Disabling field-ID reads makes the PR return the correct rows. The unrelated ID-bearing column activates the footer check and causes the failure. **Fix:** preserve root first-wins behavior in footer validation while retaining nested rejection. Add a mixed ID/name regression through the reader factory. ### Validation - **120 component tests passed**, plus Spark comparisons covering 32 read combinations. - Reviewed mapping, projection, array lifetimes, footer caching, and JNI errors. No other introduced actionable defects found. - **Build limitation:** the registry lacks DataFusion 55.1. Component probes used current sources with DataFusion 55.0 / Arrow 59.3; full Comet integration tests could not run. - **CI:** 7 passed, 12 queued/running, 9 skipped. Full review and evidence (local review report and reproduction logs). Nothing posted to GitHub. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
