yujun777 opened a new pull request, #68787: URL: https://github.com/apache/doris/pull/68787
### What problem does this PR solve? Related issue: #68742 Trace issue: https://github.com/apache/doris/issues/65418 Problem Summary: An IVM materialized view whose select list wraps an aggregate in a scalar expression, for example ```sql CREATE MATERIALIZED VIEW mv REFRESH INCREMENTAL AS SELECT k, SUM(v) * 100 AS s100 FROM t GROUP BY k; ``` is accepted and completes its first refresh, but every later incremental refresh fails with `reason=PLAN_REWRITE_FAILED, detail=IVM failed to find slot: sum(v)`. Apply merges the old MV state in the state domain and then re-applies the outer expression: `new.s100 = f(apply(old_mv.sum_v, delta.sum_v))`. That requires the MV to persist a column carrying `SUM(v)` itself, because apply reads the old state by column name from the MV physical table. `SUM`, `COUNT(expr)`, `MIN`/`MAX`, `COLLECT_LIST`/`ARRAY_AGG` and `BITMAP_UNION` keep that mergeable state in their *visible* column, and normalize drops that column as soon as an upper expression consumes it without projecting it. `AVG`, `BITMAP_UNION_COUNT` and `COUNT(*)` are unaffected because their state is already a hidden state column, which is why only some aggregates broke. A view of this shape is incremental-capable, so adding the missing state column is the fix; rejecting the shape at CREATE would cut a capability the engine already has. Fix: normalize materializes the dropped state column as a hidden pass-through carrier at the project layer that drops it, and rebinds every target that reads it, including hidden state the column pool had shared with a dropped visible column (AVG's hidden SUM). The carrier is named exactly as the delta sub-plan already names that state, so delta aggregate outputs, delta slot lookups and the write path are untouched, and shapes that already work gain no column: - `SELECT k, SUM(v) * 100` gains `__DORIS_IVM_AGG_0_SUM_COL__` (plus the existing hidden non-NULL count) - `SELECT k, SUM(v) * 100, AVG(v) * 200` gains one carrier shared by SUM and AVG's hidden SUM - `SELECT k, SUM(v) / COUNT(v)` gains one SUM carrier and one COUNT carrier for both targets - `SELECT k, SUM(v) AS s, SUM(v) * 100`, `SUM(v * 100)`, `COUNT(*) * 2`, `ROUND(AVG(v), 2)` and `BITMAP_UNION_COUNT(TO_BITMAP(v)) + 0` keep their exact previous layout Normalize additionally validates that every state column apply reads survived into the MV output, so a missing state column fails at CREATE/refresh instead of at the first incremental refresh. ### Release note None ### Check List (For Author) - Test: Regression test + Unit Test - new suite `regression-test/suites/mtmv_p0/ivm/test_ivm_agg_expr_over_agg.groovy`: hidden layout per shape via `DESC`, INSERT/UPDATE/DELETE/empty-group/resurrect windows whose values are compared against the same query over the base table, a COMPLETE refresh cross-check for every MV, and the same shape under `ivm_use_full_keys` - `fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMVTest.java`: carrier layout per aggregate function, no carrier when the visible column is projected or the value is derived, AVG's reused hidden SUM following the dropped column onto one shared carrier, and the scalar shape - all existing suites under `regression-test/suites/mtmv_p0/ivm` (102) remain green - Behavior changed: Yes — materialized views with expressions over aggregates now refresh incrementally instead of failing, and gain one hidden state column for the aggregates that need it - Does this need documentation: No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
