txwyy123 commented on issue #59787: URL: https://github.com/apache/spark/issues/59787#issuecomment-6433766327
## Benchmark Results: processNext method size before and after Measured on a 40+40 column SortMergeJoin (inner join, `maxFields=200`, AQE disabled, `splitConsumeFuncByOperator=false`). ### processNext method size | | Lines | Source chars | Est. bytecode (×1.2) | JIT compiled? | |---|---|---|---|---| | **Baseline (stock master)** | 313 | 10,221 | ~12,265 | ❌ Interpreted (~4x penalty) | | **With fix** | 197 | 5,766 | ~6,919 | ✅ C2 compiled | | **Reduction** | | **43%** | | | ### What moved out of processNext | Metric | Baseline | Fix | |---|---|---| | Column access operations in processNext | 204 | 1 | | Extraction helper methods | 0 | 6 | The fix generates 6 helper methods: - `extractStreamVars_0_0`, `extractStreamVars_0_1` (streamed-side columns, no parameter -- `streamedRow` is already a class field) - `extractJoinVars_0_0` through `extractJoinVars_0_3` (buffered-side columns, `InternalRow bufferedRow` parameter) Each helper is ~30-40 lines (~1,000-1,100 chars), well under the 8K limit. ### Test results All existing tests pass with the fix: - WholeStageCodegenSuite: 99 passed, 0 failed - JoinSuite + OuterJoinSuite + InnerJoinSuite + ExistenceJoinSuite: 669 passed, 0 failed ### Implementation approach Instead of wrapping `consume()` in a separate method (which breaks because consume references local loop variables), we store **column extraction results** in class-level mutable fields (`ctx.addMutableState`) and use `ctx.splitExpressions` to batch the extraction code into helper methods. The `consume()` code stays inline and reads from the class fields. Two new methods in `JoinCodegenSupport` and `SortMergeJoinExec`: - `genOneSideJoinVarsAsFields` -- buffered-side columns, passes `InternalRow row` as parameter to split methods - `createStreamedVarsAsFields` -- streamed-side columns, no parameter needed (streamedRow is already a class field from `genScanner`) Branch: https://github.com/txwyy123/spark/tree/wide-output-split-joins -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
