andygrove opened a new pull request, #7673: URL: https://github.com/apache/datafusion-comet/pull/7673
## Which issue does this PR close? No issue. Follow-up to #7508. **Stacked on #7508.** Until #7508 merges, this branch also carries its commits. Only the last commit (`refactor: locate FileSourceScanExec filter arguments through a shim`) is new here. I'll rebase onto `main` once #7508 lands. ## Rationale for this change #7508 rebuilds a `FileSourceScanExec` through `makeCopy` so that a runtime with extra constructor fields, such as Amazon EMR, keeps them. To find the filter arguments, it searches the constructor arguments for `requiredSchema` by identity and writes at offsets +1 and +4 from it. That handles the extra `stream` argument in Spark 4 at runtime. Comet already splits Spark 3 and Spark 4 differences into the `spark-3.x` and `spark-4.x` source sets, so that is where this layout difference belongs. The offsets are also written without checking what is there. So if a future Spark or a vendor runtime inserts a field, the scan is rebuilt with an argument in the wrong position instead of failing. ## What changes are included in this PR? - New `ShimFileSourceScanExec` objects in `spark-3.x` and `spark-4.x` give the positions of `partitionFilters` and `dataFilters` in the `FileSourceScanExec` constructor. They are 3 and 6 on Spark 3.5, and 4 and 7 on Spark 4.0, 4.1 and 4.2. I checked these against the Spark 3.4.3, 4.1.1 sources and the 4.2.0 jar. - `CometScanUtils.copyScanWithFilters` uses those positions. Before replacing each slot, it checks that the slot holds the scan's current filter list, and it throws an `IllegalStateException` that names the unexpected layout if not. - The `makeCopy` path from #7508 is unchanged, so fields that a runtime appends are still carried through. ## How are these changes tested? A new test in `CometPlanEqualitySuite` checks that the shim positions hold the scan's filter lists on whichever Spark profile it runs. The existing #7508 tests cover the fallback, `make_date` and canonicalization paths. Locally, `CometPlanEqualitySuite` and `CometExecRuleSuite` pass on the default Spark 4.1 profile and on `-Pspark-3.5`. Main and test sources compile on `-Pspark-4.2`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
