Visorgood opened a new pull request, #19017:
URL: https://github.com/apache/iceberg/pull/19017

   Closes #18953.
   
   Regression tests for SPARK-60403 (apache/spark#59653), fixed in Spark 4.2.1. 
Whole-stage codegen of a full outer `SortMergeJoinExec`, a full or build-side 
outer `ShuffledHashJoinExec` and `MergeRowsExec` called its functions by their 
bare names. Once the generated class grows past 1,000,000 characters, 
`addNewFunction` places these functions into a nested class, so the calls 
failed to compile and Spark silently fell back from whole-stage codegen.
   
   Changes:
   - `TestBase.assertCallsThroughNestedClasses`: asserts that every whole-stage 
codegen stage of a plan compiles and that the given functions are called 
through a nested class instance.
   - `TestOuterJoinCodegen`: full outer sort merge join, full outer shuffled 
hash join and a left outer shuffled hash join that builds its hash table from 
the left side, over two Iceberg tables with a 5000-field struct.
   - `TestMerge.testMergeWithWideStructCallsFunctionsThroughNestedClasses`: 
MERGE with matched, not matched and not matched by source clauses (a full outer 
join) into a table with a 1000-field struct, for copy-on-write and 
merge-on-read.
   
   Notes for review:
   - The upstream test puts a sliding-window aggregation above the join. On 
Spark 4.2 that layout doesn't fail without the fix: the join's functions land 
in the same nested class as their caller (`doAggregateWithKeys`), so the bare 
calls compile. On 4.2.0 it fails only on SPARK-60398 (`ExpandExec`). Writing a 
wide struct with no aggregation above the join keeps the calls in `processNext` 
of the outer class, where they fail without the fix.
   - Whole-stage codegen skips plans with more fields than 
`spark.sql.codegen.maxFields` (100 by default, nested fields included), so the 
tests raise that limit. They also disable `spark.sql.codegen.fallback`, so a 
compile error fails the test instead of falling back.
   - The struct widths (5000 for the joins, 1000 for MERGE) depend on how much 
code Spark generates. 2500 and 500 were not enough, so a future Spark version 
may need different values.
   - The MERGE test runs only for the Parquet parameters, since the generated 
code doesn't depend on the file format.
   
   Testing:
   - Passes on Spark 4.2.1 (`TestOuterJoinCodegen` ~24 s; MERGE ~9 s 
copy-on-write, ~3 s merge-on-read).
   - With `spark42 = "4.2.0"` the new tests fail with "A method named ... is 
not declared in any enclosing class".
   
   Co-Authored with Claude Opus 5.5.
   
   cc @manuzhang


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to