darpan-e6 opened a new pull request, #4970: URL: https://github.com/apache/calcite/pull/4970
## Jira Link [CALCITE-7551](https://issues.apache.org/jira/browse/CALCITE-7551) ## Changes Proposed Several optimizer and `RelBuilder`-level transformations inline a lower-projection expression into multiple positions of an upper expression without checking whether the inlined expression is non-deterministic. When the lower expression is non-deterministic (e.g. `RAND()`), this silently turns one evaluation into many, changing query semantics. ### Reproduction ```sql SELECT a, a + 1 AS b FROM (SELECT rand() AS a) ``` Before the fix, `SqlToRelConverter` produces: ``` LogicalProject(A=[RAND()], B=[+(RAND(), 1)]) LogicalValues(tuples=[[{ 0 }]]) ``` The two `RAND()` calls are evaluated independently, so `B - A` is no longer always `1`. After the fix the inner projection is preserved and `A`, `B` share a single evaluation: ``` LogicalProject(A=[$0], B=[+($0, 1)]) LogicalProject(A=[RAND()]) LogicalValues(tuples=[[{ 0 }]]) ``` ### Affected code paths and fixes | Component | Mechanism | Fix | |---|---|---| | `RelOptUtil.pushPastProjectUnlessBloat` | `pushShuttle(Project)` substitutes every `RexInputRef` with the underlying projection. Only `RexOver` was guarded; determinism was not. | Count `RexInputRef` uses per bottom slot; return `null` if any non-deterministic bottom expression is referenced more than once. Covers `ProjectMergeRule`, `FilterProjectTransposeRule`'s helper call, and the `RelBuilder` bloat merge. | | `FilterProjectTransposeRule` | Calls `pushPastProjectUnlessBloat` AND re-emits the original projection above the new filter — so even with the helper guarded, `Project(rand() AS a)` on top of `Filter(rand() > ...)` splits one evaluation into two. | Skip when any projected expression is non-deterministic, parallel to the existing `containsOver` skip. | | `JoinProjectTransposeRule` | Uses `RexProgramBuilder.mergePrograms` + `mergedProgram.expandLocalRef(condition)`. `expandLocalRef` flattens local refs back into a `RexNode` tree, inlining the underlying expression at every occurrence. | Skip non-deterministic projections, parallel to the existing `containsOver` skip on each side. | | `SemiJoinProjectTransposeRule` | Same `mergePrograms` + `expandLocalRef` pattern. | Same skip. | The fix uses `RexUtil.isDeterministic(...)`, which correctly distinguishes operators that may return different values across call sites within a single statement (`RAND` overrides to `false`) from operators that are non-deterministic only across executions (`CURRENT_TIMESTAMP`, marked `isDynamicFunction = true` but still `isDeterministic = true`). The latter is safe to duplicate and is unaffected by this change. ### Tests - `RelOptRulesTest.testProjectMergeShouldIgnoreNonDeterministic` - `RelOptRulesTest.testFilterProjectTransposeShouldIgnoreNonDeterministic` - `RelOptRulesTest.testJoinProjectTransposeShouldIgnoreNonDeterministic` - `RelOptRulesTest.testSemiJoinProjectTransposeShouldIgnoreNonDeterministic` - `SqlToRelConverterTest.testRandNotDuplicatedInProjectionMerge` Each follows the existing `.checkUnchanged()` / `.ok()` fixture pattern (parallel to `testProjectMergeShouldIgnoreOver`) and verifies that the rule (or the `SqlToRelConverter` build path) does not produce a plan with duplicated `RAND()` calls. ### Confirmed unaffected (verified by additional probing during investigation) `CalcMergeRule` (RexProgram preserves CSE via local refs), `ProjectReduceExpressionsRule`, `ProjectValuesMergeRule`, `ProjectFilterTransposeRule`, `ProjectJoinTransposeRule`, `ProjectCorrelateTransposeRule`, `AggregateProjectMergeRule`, `FilterSetOpTransposeRule` (textual duplication only — each row flows through one branch of a `UNION ALL`, so per-row evaluation count is unchanged), `ProjectToWindowRule`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
