shiliang opened a new pull request, #70569: URL: https://github.com/apache/doris/pull/70569
### Background `array_map` builds a per-element lambda block for nested lambdas. Outer lambda arguments that remain visible on the `LambdaExecutionContext` stack used to be copied into that block whether or not the nested body read them. SlotRefs are collected separately; name-bound lambda arguments are resolved through `resolve_column_position`. ### What problem does this PR solve? Issue Number: close #69918 Related PR: None PR Type: Bug Fix Problem Summary: A nested `array_map` copied every visible outer lambda argument into the inner lambda block once per element, including outer arrays the body never reads. For `array_map(x -> array_map(y -> y + 1, x), arr_of_arrays)` this made memory quadratic in array length. Restrict capture to bindings the body actually resolves by name (innermost shadowing), and rename the pad column from `temp` to `__DORIS_LAMBDA_BLOCK_PAD__` so it cannot collide with a user argument named `temp`. ### Details - **Change map:** ~30 lines production (`varray_map_function.cpp`, `lambda_execution_context.h`), ~105 lines unit test. - **Frequency:** Any nested `array_map` whose outer arguments include unused wide arrays; worse as array length grows. - **Root cause:** `collect_visible_binding_column_positions` inserted every visible binding position into `required_input_column_ids`, so unused outer arrays were expanded with `insert_indices_from` / `clone_resized` per inner element. - **Why existing tests missed it:** Nested tests asserted result correctness for referenced captures, not that unused outer arguments stayed unmaterialized. - **Fix:** When `bind_by_name`, walk the lambda body and collect only `is_column_ref` names that `resolve_column_position` finds; keep SlotRef collection unchanged. Pad unused sparse slots with `__DORIS_LAMBDA_BLOCK_PAD__`. - **Other paths and branches:** Legacy `bind_by_name=false` path unchanged. `array_sorted_merge` still uses `collect_visible_bindings` by name on its private block. ### Testing - Added BE UT `NestedLambdaDoesNotMaterializeUnusedOuterArgument`: asserts the inner lambda block has only the `y` column while still returning correct `y+1` results with a large unused outer array. - Existing nested `array_map` UTs in `array_map_function_test.cpp` remain the regression net for referenced outer captures and shadowing (to be re-run in CI `run beut`). ### Notes for Reviewers Start at `ArrayMapFunction::execute` where `required_input_column_ids` is built; the new helper is `_collect_referenced_binding_column_positions`. ### Release note Nested `array_map` no longer materializes unused outer lambda arguments into each inner element, avoiding quadratic memory growth on large arrays. Query results are unchanged. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: unit test added; BE UT to be run in CI ### AI Disclosure - Tool: Cursor Agent - Generated: code, unit test, and this PR description - Human review: Please review capture selection, pad rename, and the new UT against #69918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
