yujun777 opened a new pull request, #70804: URL: https://github.com/apache/doris/pull/70804
### Background An incremental aggregate materialized view applies signed changes from its base tables to the counts it already stores. Runtime guards reject a merge that would make a count negative. A refresh request that allows fallback can rebuild the view when the stored state cannot be used to apply the change. ### What problem does this PR solve? Issue Number: None Related PR: #62606 (introduced the aggregate guards), #69772 (uses the same recovery path for expired stream offsets) PR Type: Bugfix Problem Summary: An incremental refresh that calculates a negative count fails even when the request allows fallback. This can be reproduced by moving facts to an already populated group through an excluded dimension, then deleting more facts than the view has recorded for that group. The refresh now rebuilds the view and its stream baselines when the request allows fallback. ### Details - **Frequency:** Reproduced with SQL when the delete delta for a group exceeds the count stored by the view. Production frequency is unknown. - **Root cause:** The aggregate assertions returned ordinary query errors that IVM's execution failure classifier did not recognize. The incremental attempt propagated the error and the refresh task never reached the fallback path. - **Why existing tests missed it:** `IvmIncrRefreshManagerTest` covers other recognized executor errors. `test_ivm_minmax_runtime_fallback` reaches the fallback path through a MIN/MAX boundary guard and does not make an aggregate count negative. - **Fix:** Tag the shared non-negative count assertion and classify it as `NEGATIVE_AGGREGATE_COUNT`. Use the existing direct-to-`COMPLETE` route to recompute view state and reset the stream baselines together. - **Other paths and branches:** The shared guard covers the group count, hidden non-null counts, and `COUNT(expr)` in both deployment modes. The Apache Doris 3.1, 4.0, 4.1, and 4.2 release branches checked do not contain this classifier. - **Behavior change:** `INCREMENTAL FALLBACK` and `AUTO` try `COMPLETE` after this guard trips. Strict `INCREMENTAL` reports `NEGATIVE_AGGREGATE_COUNT` and includes the assertion detail. ### Testing - Reproduced `negative group count` with an unset fallback reason on a local cluster running the unfixed FE. A second refresh with `INCREMENTAL FALLBACK` failed on the same assertion. - `./build.sh --fe` completed with zero FE Core Checkstyle violations. - `test_ivm_negative_aggregate_count_fallback` passed on the local shared-nothing cluster. Strict incremental failed with the classified reason and preserved the rows. Both fallback-enabled modes finished as `COMPLETE`, followed by a successful incremental refresh. - All 129 suites under `mtmv_p0/ivm` passed on the local shared-nothing cluster. - All 548 selected FE IVM and materialized view unit tests passed. ### Release note Incremental aggregate materialized view refreshes that allow fallback now attempt a complete refresh if applying a change would make a count negative. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
