matanper opened a new issue, #69918: URL: https://github.com/apache/doris/issues/69918
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues) and found no similar issues. ### Version master (checked at its current head), branch-4.1 ### What's wrong? A nested lambda copies every outer lambda argument that is visible to it into its own block, once per element it evaluates, including outer arguments it never reads. In `FunctionArrayMap` (`be/src/exprs/lambda_function/varray_map_function.cpp`), the required input columns are collected like this: ```cpp children[0]->collect_slot_column_ids(required_input_column_ids); context->lambda_execution_context().collect_visible_binding_column_positions( required_input_column_ids); ``` `LambdaExecutionContext::collect_visible_binding_column_positions` (`lambda_execution_context.h`) adds the column position of every argument binding in every visible frame, whether or not the lambda body references it. Each of those columns is then materialized for every nested element (`clone_resized` / `insert_indices_from` over `captured_source_row_indices`). When an outer lambda's argument is itself an array, and the inner lambda iterates over that array, the inner block holds a copy of the whole outer array for each of its elements. The work and memory are therefore quadratic in the array length: ```sql -- x is an array of N elements; the inner lambda never reads x itself, only its elements, -- but every inner row carries a copy of x SELECT array_map(x -> array_map(y -> y + 1, x), arr_of_arrays) FROM t; ``` From reading the code: with N = 10,000 BIGINT elements, one batch holds about N² values (several hundred MB) for an expression that needs N. #66590 (`cb0f7a434bd`) bounds the size of each batch, but each batch still carries the copies. ### What you expected to happen A nested lambda materializes only the outer bindings its subtree references. Bindings it does not reference stay unmaterialized, as the `ColumnNothing` / placeholder path already allows for unrelated input columns. ### How to reproduce Run the query above on a table whose array column has a few thousand elements per row, and compare the peak memory and time with the equivalent single-level `array_map`. ### Anything else? Restricting the copy is subtle. We tried matching referenced names against every visible frame, and two regressions came out of it: - a name that a closer lambda re-declares (`(a, b) -> array_map(b -> array_map(z -> z + b, ...), ...)`) must keep the binding that `resolve_column_position` returns, not every binding with that name; - the `"temp"` UInt8 placeholder for an unmaterialized position can be found by name ahead of a real lambda argument called `temp`. So the referenced set should be resolved per name to the innermost binding, and placeholders should carry a name no argument can have. Related: #68977 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
