Doris-Breakwater commented on issue #70277: URL: https://github.com/apache/doris/issues/70277#issuecomment-6654232831
Initial triage: **high-confidence snapshot correctness defect supported by the pinned source**, with a credible join-IVM consequence. The SQL reproduction and an IVM refresh have **not been executed in this investigation**, so the predicted missing rows remain to be confirmed at runtime. This assessment is for `72e91896bbdd8760f8bd4e488e5672406b40ce52`; it does not establish affected release branches. **Verified code path** 1. Snapshot reconstruction for changed non-duplicate-key partitions explicitly requests `MIN_DELTA`, independently of the stream's configured type. It unions a historical-TSO-bounded base scan with changelog before images. The changelog branch retains only `DELETE` and `UPDATE_BEFORE`. See [snapshot reconstruction](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NormalizeOlapTableStreamScan.java#L328-L344) and [operation filter](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NormalizeOlapTableStreamScan.java#L223-L229). 2. The offset conversion preserves the intended boundary: the base branch includes commits through the consumption TSO, while the changelog starts after it. BE applies these as half-open TSO predicates. A replacement row committed after the offset fails the base predicate; the old MoW row is not recovered merely by filtering the current base image. See [offset conversion](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/catalog/stream/OlapTableStreamWrapper.java#L345-L365) and [BE TSO filtering](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/src/exec/scan/olap_scanner.cpp#L350-L377). 3. In `UPDATE_BEFORE_AFTER`, the reader emits neither row when the first before image equals the final after image. This happens before the snapshot's operation filter can recover anything. `DELETE → APPEND` also maps to `UPDATE_BEFORE_AFTER`, so the delete/reinsert scenario is supported by the same mechanism. See [suppression](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/src/storage/iterator/block_reader.cpp#L318-L323) and [transition matrix](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/src/storage/iterator/changelog_block_reader_utils.h#L42-L71). The contract mismatch is that **zero net change does not imply that no before image is needed to reconstruct the offset snapshot**. This also covers a value round trip such as `a → b → a`, not only an immediately identical upsert. Existing [reader tests](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/test/storage/iterator/block_reader_change_next_block_test.cpp#L660-L707) intentionally expect such net-zero chains to disappear from ordinary MIN_DELTA output; those tests were inspected, not run. The equality check conservatively retains updates for unavailable/all-NULL before images or unsupported comparisons, so the omission is conditional; the supplied non-NULL VARCHAR example fits the supported comparison case. **IVM impact: supported call path, unverified outcome** [The join handler](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmJoinDeltaHandler.java#L116-L135) uses the other side's pre-snapshot for a left-side delta. A focused regression should establish a baseline, rewrite a matching right-side MoW row identically, and insert a matching left-side row before incremental refresh. In the contribution `ΔL JOIN R_pre`, the missing right-side snapshot row can suppress the new match, while `ΔR` is empty. An identical rewrite alone does not demonstrate an incorrect MV refresh. Capture the actual rewritten plan to confirm which side supplies the pre-snapshot. **Missing evidence and next steps** - Run both supplied reproductions on the pinned build, waiting for writes to become visible. Capture actual base/stream/snapshot rows, FE and BE build identifiers, deployment mode, `SHOW CREATE TABLE`, `SHOW CREATE STREAM`, stream consumption state before/after, and `EXPLAIN VERBOSE` for the snapshot. The issue already supplies a useful version and minimal SQL; the main gap is executed evidence. Query IDs and relevant FE/BE logs are useful if execution diverges from this path. - Preserve per-key folding for snapshot reconstruction while retaining the first required before image even for equal endpoints, using a snapshot-specific reader contract. Keep ordinary net-change suppression semantics. A direct switch to `DETAIL` plus the existing operation filter is insufficient: `a → b → a` exposes intermediate before images and can add extra rows unless reconstruction selects the correct first image per key. Changing the stream type also does not bypass the hard-coded snapshot scan type. - Add snapshot regressions for identical upsert, delete/reinsert, and value round trips, alongside changed-value, insert-only, delete-only, and untouched-row controls. Check repeated snapshot reads do not advance consumption. Extend coverage to stream types sharing this reconstruction path and to small batches/compaction. - Add the join-IVM case above and compare incremental results against a fresh base-table evaluation with both `enable_materialized_view_rewrite=false` and `enable_dml_materialized_view_rewrite=false`. Confirm incremental execution rather than a complete-refresh fallback. Suggested routing: Table Stream/Nereids planning and BE changelog reader maintainers, with IVM maintainers reviewing the integration regression. No code changes were made. Breakwater-GitHub-Analysis-Slot: slot_9bbb44509cce -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
