andygrove opened a new pull request, #7673:
URL: https://github.com/apache/datafusion-comet/pull/7673

   ## Which issue does this PR close?
   
   No issue. Follow-up to #7508.
   
   **Stacked on #7508.** Until #7508 merges, this branch also carries its 
commits. Only the last commit (`refactor: locate FileSourceScanExec filter 
arguments through a shim`) is new here. I'll rebase onto `main` once #7508 
lands.
   
   ## Rationale for this change
   
   #7508 rebuilds a `FileSourceScanExec` through `makeCopy` so that a runtime 
with extra constructor fields, such as Amazon EMR, keeps them. To find the 
filter arguments, it searches the constructor arguments for `requiredSchema` by 
identity and writes at offsets +1 and +4 from it. That handles the extra 
`stream` argument in Spark 4 at runtime. Comet already splits Spark 3 and Spark 
4 differences into the `spark-3.x` and `spark-4.x` source sets, so that is 
where this layout difference belongs.
   
   The offsets are also written without checking what is there. So if a future 
Spark or a vendor runtime inserts a field, the scan is rebuilt with an argument 
in the wrong position instead of failing.
   
   ## What changes are included in this PR?
   
   - New `ShimFileSourceScanExec` objects in `spark-3.x` and `spark-4.x` give 
the positions of `partitionFilters` and `dataFilters` in the 
`FileSourceScanExec` constructor. They are 3 and 6 on Spark 3.5, and 4 and 7 on 
Spark 4.0, 4.1 and 4.2. I checked these against the Spark 3.4.3, 4.1.1 sources 
and the 4.2.0 jar.
   - `CometScanUtils.copyScanWithFilters` uses those positions. Before 
replacing each slot, it checks that the slot holds the scan's current filter 
list, and it throws an `IllegalStateException` that names the unexpected layout 
if not.
   - The `makeCopy` path from #7508 is unchanged, so fields that a runtime 
appends are still carried through.
   
   ## How are these changes tested?
   
   A new test in `CometPlanEqualitySuite` checks that the shim positions hold 
the scan's filter lists on whichever Spark profile it runs. The existing #7508 
tests cover the fallback, `make_date` and canonicalization paths.
   
   Locally, `CometPlanEqualitySuite` and `CometExecRuleSuite` pass on the 
default Spark 4.1 profile and on `-Pspark-3.5`. Main and test sources compile 
on `-Pspark-4.2`.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to