Mrhs121 opened a new pull request, #5396: URL: https://github.com/apache/calcite/pull/5396
## Jira Link [CALCITE-7924](https://issues.apache.org/jira/browse/CALCITE-7924) ## Changes Proposed ### Problem This issue was discovered while compiling SQL in Feldera and reproduced using Calcite alone. Expression-lineage inference over `UNION ALL` can construct the Cartesian product of the possible origins of each referenced column. With N branches and K referenced columns, this can produce N^K candidate expressions and cause excessive allocation and GC during planning. Both scalar expressions and window expressions in `Project`/`Calc` are affected. ### Proposed solution Bound candidate construction in the shared `RelMdExpressionLineage.createAllPossibleExpressions` helper, before it copies any candidate `RexNode`: - Check the product of replacement-set sizes for referenced input columns against a configurable work budget. Repeated occurrences of a reference share a substitution, and unused mapping keys do not add a dimension. - Use division to check the product without integer overflow. Count construction attempts before deduplication, rather than the size of the final result set. - If the budget would be exceeded, return `null` (unknown metadata), never a truncated set presented as complete lineage. Keep the existing expansion and results within the budget, including small window-expression cases; this is not a blanket rejection of window lineage. - Introduce `calcite.metadata.expression.lineage.max.expansions`, defaulting to `10000`. `-1` restores unlimited expansion, and `0` disables expansions involving input references. Constants require no expansion and remain unaffected. Values below `-1` fall back to the default. - Document the bounded behavior in the metadata contract. The shared helper also covers all-predicates inference. The budget is per expansion, not query-wide. It does not bound metadata traversal or the size of an individual expression. The default threshold and configuration mechanism are open for discussion. ### Reproducer and regression coverage Add a parameterized, storage-free program generator with adjustable `UNION ALL` branch count, referenced-column count, and scalar/window expression type. The tests exercise both `Project` and `Calc`, budget boundaries, unlimited/zero budgets, duplicate candidates, repeated and unused references, overflow prevention, metadata cache invalidation, and all-predicates inference. For example, `ExpressionLineageProgram.rel(16, 4, true)` produces a plan whose computed-column lineage expands to 65,536 candidates without the limit. With the proposed default it returns unknown before enumerating those candidates. Smaller cases still return complete lineage, and querying a passthrough column remains supported after a budget rejection. ### Observed impact In the pure Calcite 32-branch, four-column window case, median lineage-call time decreased from 2,026.93 ms to 10.97 ms, with peak process RSS decreasing from 879.31 MiB to 169.55 MiB. The 8-branch case remains within budget and retains all 4,096 candidates. For the full anonymized downstream SQL, changing only the Calcite budget implementation reduced SQL-to-Rust generation from 402.32 s to 4.83 s and peak process RSS from 8.35 GiB to 522.48 MiB. This comparison does not include native Rust compilation. The improvement comes from bounding optional metadata inference, not from computing the same large candidate set faster. The reproducer and detailed measurements are in the linked Jira issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
