yujun777 opened a new pull request, #70561:
URL: https://github.com/apache/doris/pull/70561
### What problem does this PR solve?
Problem Summary:
Incremental outer-join refresh currently checks the null-producing side's
pre- and post-refresh snapshots to determine whether to retract or insert a
NULL-extended row. It builds these two existence checks even when the complete
join key is proven unique.
For a unique matching key, the pre/post row counts are each either zero or
one. Summing the complete signed delta therefore gives `net_count = post_count
- pre_count`, and the NULL repair factor is exactly `-net_count`. This change
retains the original detail events and derives the repair directly from the
delta, eliminating both repair ANTI JOINs and their snapshot subtrees.
The simplified plan below shows only the contribution from changes to B in
`A LEFT JOIN B`. The retained A snapshot and other refresh contributions still
require their normal snapshot reads.
```diff
INNER JOIN (complete join key)
|-- A_post snapshot
`-- UNION ALL
|-- delta_B detail events
- |-- PROJECT (NULL payload, factor = -1)
- | `-- LEFT ANTI JOIN
- | |-- FILTER (has_positive)
- | | `-- AGGREGATE (key, MAX positive/negative flags)
- | | `-- delta_B
- | `-- B_pre snapshot
- `-- PROJECT (NULL payload, factor = +1)
- `-- LEFT ANTI JOIN
- |-- FILTER (has_negative)
- | `-- AGGREGATE (key, MAX positive/negative flags)
- | `-- delta_B
- `-- B_post snapshot
+ `-- PROJECT (NULL payload, factor = -net_count)
+ `-- FILTER (net_count <> 0)
+ `-- AGGREGATE (key, SUM(dml_factor) AS net_count)
+ `-- FILTER (join keys IS NOT NULL)
+ `-- delta_B
```
The specialization uses the existing uniqueness trait of the original
relation, accepts direct slot equality keys, and supports LEFT and RIGHT OUTER
JOIN. Ordinary value updates keep their `-old/+new` detail events. Nonunique
keys, unproven expression keys, residual predicates, and FULL OUTER JOIN keep
their existing rewrite paths. NULL keys are excluded only from repair
candidates because ordinary equality cannot match them. The internal aggregate
is generated during delta rewriting, after MV normalization.
### Release note
Reduce snapshot existence checks during incremental outer-join refresh when
the null-producing side's complete join key is proven unique.
### Check List (For Author)
- Test:
- FE and ASAN BE builds, plus separate Checkstyle validation.
- 194 focused FE tests: IvmJoinDeltaHandlerTest, IvmDeltaRewriterTest,
IvmNormalizeUniqKeysMTMVJoinTest, and CreateMTMVCommandTest.
- Regression comparison: test_ivm_unique_key_null_repair and
test_ivm_outer_join_1 through test_ivm_outer_join_6 (7 suites passed).
- New golden output generated with the runner and inspected: 8 change
stages, 32 empty signed bag differences against full base-table queries.
- Actual chained aggregate-MV analyzed plans: LEFT/RIGHT use SUM repair
without repair ANTI JOINs; incomplete-key and residual cases retain their
existence checks.
- Behavior changed: Yes. Eligible refresh plans derive NULL repair from the
signed delta instead of snapshot ANTI JOINs; query results are preserved.
- Does this need documentation: No. This is an internal refresh-plan
optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
