yujun777 opened a new pull request, #70804:
URL: https://github.com/apache/doris/pull/70804

   ### Background
   
   An incremental aggregate materialized view applies signed changes from its 
base tables to the counts it already stores. Runtime guards reject a merge that 
would make a count negative. A refresh request that allows fallback can rebuild 
the view when the stored state cannot be used to apply the change.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #62606 (introduced the aggregate guards), #69772 (uses the same 
recovery path for expired stream offsets)
   PR Type: Bugfix
   
   Problem Summary:
   
   An incremental refresh that calculates a negative count fails even when the 
request allows fallback. This can be reproduced by moving facts to an already 
populated group through an excluded dimension, then deleting more facts than 
the view has recorded for that group. The refresh now rebuilds the view and its 
stream baselines when the request allows fallback.
   
   ### Details
   
   - **Frequency:** Reproduced with SQL when the delete delta for a group 
exceeds the count stored by the view. Production frequency is unknown.
   - **Root cause:** The aggregate assertions returned ordinary query errors 
that IVM's execution failure classifier did not recognize. The incremental 
attempt propagated the error and the refresh task never reached the fallback 
path.
   - **Why existing tests missed it:** `IvmIncrRefreshManagerTest` covers other 
recognized executor errors. `test_ivm_minmax_runtime_fallback` reaches the 
fallback path through a MIN/MAX boundary guard and does not make an aggregate 
count negative.
   - **Fix:** Tag the shared non-negative count assertion and classify it as 
`NEGATIVE_AGGREGATE_COUNT`. Use the existing direct-to-`COMPLETE` route to 
recompute view state and reset the stream baselines together.
   - **Other paths and branches:** The shared guard covers the group count, 
hidden non-null counts, and `COUNT(expr)` in both deployment modes. The Apache 
Doris 3.1, 4.0, 4.1, and 4.2 release branches checked do not contain this 
classifier.
   - **Behavior change:** `INCREMENTAL FALLBACK` and `AUTO` try `COMPLETE` 
after this guard trips. Strict `INCREMENTAL` reports `NEGATIVE_AGGREGATE_COUNT` 
and includes the assertion detail.
   
   ### Testing
   
   - Reproduced `negative group count` with an unset fallback reason on a local 
cluster running the unfixed FE. A second refresh with `INCREMENTAL FALLBACK` 
failed on the same assertion.
   
   - `./build.sh --fe` completed with zero FE Core Checkstyle violations.
   - `test_ivm_negative_aggregate_count_fallback` passed on the local 
shared-nothing cluster. Strict incremental failed with the classified reason 
and preserved the rows. Both fallback-enabled modes finished as `COMPLETE`, 
followed by a successful incremental refresh.
   
   - All 129 suites under `mtmv_p0/ivm` passed on the local shared-nothing 
cluster.
   - All 548 selected FE IVM and materialized view unit tests passed.
   
   ### Release note
   
   Incremental aggregate materialized view refreshes that allow fallback now 
attempt a complete refresh if applying a change would make a count negative.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No
   - Pre-submission status: Ready
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to