anshulbaliga7 commented on PR #58077: URL: https://github.com/apache/spark/pull/58077#issuecomment-5668488066
Thanks for the review @cloud-fan, have addressed all four: - LHS binding (P1): the multi-column codegen branch now generates the LHS through `child.genCode(ctx)` like every other expression so it consumes the operator's bound `ctx.currentVars` (e.g. streamed ++ build variables of a residual join condition) instead of interpreting the bound child against ctx.INPUT_ROW. Only the evaluated struct value is passed to the evaluator. Added an E2E regression with a whole-stage code-generated hash join whose NOT IN LHS references both join sides, asserting the join runs inside `WholeStageCodegenExec` with `InSubqueryExec` in its condition. - Nested plans in references (P2): `MultiColumnInSubqueryEvaluator` no longer owns any expression. It holds only the result rows, their field types and the legacy empty-list flag, so nothing reachable through the LHS (scalar subquery, nested `InSubqueryExec` or anything added later) can enter task closures. The `ScalarSubquery` literalization and the manual Nondeterministic registration loop are gone; those nodes now register their own state through normal codegen. Added a regression with both a `ScalarSubquery` and a nested `InSubqueryExec` in the LHS that asserts no SparkPlan or expression holding a PlanExpression lands in `ctx.references` and executes the generated predicate. - Serialization test (P2): the round-trip test now calls eval on the deserialized instance across TRUE / UNKNOWN / FALSE / null-field / null-struct cases. - Dispatch caching (P3): `isMultiColumn` is computed once per instance and used in nullable, updateResult, eval and `doGenCode`. Two additional hardenings while re-verifying the earlier rounds: the SQL cases that used an inner join with a one-sided IN were being pushed into a Filter by `PushPredicateThroughJoin` and rewritten to a semi/anti join so they never reached `InSubqueryExec`. Converted them to FULL OUTER JOIN and added a shared plan assertion to every SQL case. Also added a regression with implicitly widened LHS/RHS types (SMALLINT/INT vs INT/BIGINT), since the evaluator now derives its orderings from the subquery output types. 14/14 SPARK-58481 tests, the full SubquerySuite, both dpp suites and the in-subquery golden files pass locally. Can you PTAL again? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
