txwyy123 commented on issue #59787: URL: https://github.com/apache/spark/issues/59787#issuecomment-6409280213
## Update: Revised approach implemented and passing The initial `addNewFunction` wrapping approach (wrapping `consume()` in a class-level method) failed because `consume()` generates Java code that references local loop variables. Moving it to a separate method makes those variables inaccessible. ### New approach: field-based variable storage with `splitExpressions` Instead of moving `consume()` to a separate method, we split the **column extraction code** by storing extracted values in class-level mutable fields instead of local variables. This lets `ctx.splitExpressions` batch the extraction into helper methods, each small enough for JIT compilation. **Implementation:** - Added `genOneSideJoinVarsAsFields` to `JoinCodegenSupport` — like `genOneSideJoinVars` but uses `addMutableState` for isNull/value fields - `SortMergeJoinExec.doProduce` uses this for the buffered side, with extraction code injected before the condition check - `splitExpressions` automatically batches the per-column extraction code into helper methods when the total exceeds the JIT threshold **Key details:** - Outer joins need `isNull` fields even for non-nullable columns (the column becomes nullable when there's no match) - The row is passed as a parameter to split methods, so `row.isNullAt(i)` / `row.getInt(i)` calls work correctly - ExprCode objects have empty `.code` (extraction pre-evaluated), so existing `evaluateVariables`/`splitVarsByCondition` machinery treats them as no-ops **Test results:** - WholeStageCodegenSuite: 99 passed, 0 failed - JoinSuite: 59 passed - OuterJoinSuite: 96 passed - InnerJoinSuite + ExistenceJoinSuite: 514 passed - New tests for wide (40+40 col) SMJ and BHJ: passing Branch: https://github.com/txwyy123/spark/tree/wide-output-split-joins -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
