yujun777 opened a new pull request, #70630:
URL: https://github.com/apache/doris/pull/70630

   ### Background
   
   Incremental materialized views maintain outer joins by writing joined-row 
changes and repairs for rows whose nullable side is filled with NULL. 
Unique-key views apply these writes through merge-on-write keys and sequence 
numbers.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #62606 (introduced the bug)
   PR Type: Bugfix
   
   Problem Summary:
   
   Incremental refresh can lose a padded NULL row when the nullable-side join 
subtree emits cancelling positive and negative events. Order NULL-row repairs 
by their operation sign so reinsertion wins when both repairs target the same 
row.
   
   For a view over `l LEFT JOIN (a JOIN b)`, updating the outer join key in `a` 
and deleting its matching `b` row in the same refresh window produced:
   
   | Result | Rows |
   |---|---|
   | Before the fix, one observed incremental refresh | `(1,10,NULL)` |
   | With the fix, incremental and complete refresh | `(1,10,NULL), 
(2,20,NULL)` |
   
   ### Details
   
   - **Frequency:** Intermittent when changes to both inputs of a nullable-side 
join produce cancelling rows within one refresh window.
   - **Root cause:** `paddingSequence()` ignored the repair sign. Both the 
NULL-row deletion and reinsertion used the same sequence, so merge-on-write 
could retain the deletion according to arrival order.
   - **Why existing tests missed it:** `IvmJoinDeltaHandlerTest` checked the 
repair sign and sequence column name without checking their ordering. 
`test_ivm_refresh_dry_run` masks the sequence value, while final-row checks can 
pass when reinsertion happens to arrive last.
   - **Fix:** Reuse the existing signed sequence encoder in both the general 
repair and equality-event paths. Preserve the refresh prefix and delta index, 
and distinguish repairs using the existing operation bit without adding scans 
or joins.
   - **Other paths and branches:** LEFT, RIGHT and FULL joins share these 
repair producers. BIGINT and LARGEINT encodings are covered; the aggregate 
final-sequence path and append-only detail model are unchanged. The handler 
introduced by #62606 is absent from branch-3.1, branch-4.0, branch-4.1 and 
branch-4.2.
   - **Behavior change:** DRY RUN now exposes an even sequence for negative 
NULL-row repairs and a larger odd sequence for positive repairs; its result 
schema is unchanged.
   
   ### Testing
   
   - Three targeted unit tests fail on the unpatched code at the 
repair-sequence assertions.
   - The IVM regression directory passed locally: 117 suites, zero failures and 
zero skips, including the new `test_ivm_outer_join_null_repair_sequence` case 
for equality events, residual predicates, RIGHT JOIN and FULL JOIN.
   - IVM FE unit tests and create/show/drop materialized-view tests were run 
locally; the final combined verification is in progress.
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No
   - Pre-submission status: Partially verified: final combined FE unit-test run 
is in progress
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to