danielwei816 opened a new pull request, #7508: URL: https://github.com/apache/datafusion-comet/pull/7508
## Which issue does this PR close? Closes #7393. ## Rationale for this change We hit a `NoSuchMethodError` on EMR 8.1.0 when Spark canonicalizes a Comet scan. EMR adds two fields to `FileSourceScanExec`, so its `copy()` signature differs from the one Comet was compiled against. ## What changes are included in this PR? Use `makeCopy()` with the scan's existing runtime fields, changing only the partition and data filters. This covers both canonicalization and scan fallback, and preserves any extra fields. It follows the approach used in #207. ## How are these changes tested? - Added a regression with two extra scan fields. It fails with the old copy call and passes with this change. - `CometPlanEqualitySuite`: 4 tests passed; the Spark 4.2-only test was skipped on Spark 4.1. - `CometExecSuite` partition-pruning tests: 37 passed. - Public ARM64 EMR 8.1.0 image: reproduced the original error, verified the current helper preserves both extra fields, and applied the same copy fix to Comet 1.1.0. The query then canonicalized and cached successfully, returning the expected 90 rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
