morningman commented on PR #69778:
URL: https://github.com/apache/doris/pull/69778#issuecomment-6471694340
### When does the problem this PR fixes show up? Three examples
Rows go missing only when three things hold at the same time:
1. **The partition has no stream offset.** An IVM MV's internal stream
records an offset for a partition only when it consumes the partition with data
in it, so two kinds of partitions have none:
- a partition added after the MV was built (dynamic or auto partitioning,
`ALTER TABLE ... ADD PARTITION`);
- a partition that already existed but was still empty when the MV was
built (future partitions pre-created by `dynamic_partition.end`, a LIST
partition that is not in use yet).
2. **Its rows wait longer than `changelog.ttl_seconds` before the first
refresh that reads the partition** (the TTL is one day when the property is not
set). Either the refresh interval is longer than the TTL, or refreshes stopped
for a while: the job was paused, refreshes kept failing, or nobody refreshed a
manually refreshed MV.
3. **Nothing else in that refresh trips the expiry check.** If a partition
that does have an offset also has new data and its offset has expired, the
whole incremental read is rejected. On a unique-key table that fails the
refresh today, and with #69772 a refresh that may fall back runs COMPLETE,
which happens to repair the MV. Either way it is loud. A table partitioned by
time and written only in its latest partitions meets this condition naturally,
because its old partitions do not change.
#### Example 1: daily partitions, refreshed once a week
```sql
CREATE TABLE orders (
order_id BIGINT NOT NULL,
dt DATE NOT NULL,
amount DECIMAL(12, 2)
)
UNIQUE KEY(order_id, dt)
PARTITION BY RANGE(dt) ()
DISTRIBUTED BY HASH(order_id) BUCKETS 8
PROPERTIES (
"enable_unique_key_merge_on_write" = "true",
"changelog.enable" = "true",
"changelog.need_historical_value" = "true",
"dynamic_partition.enable" = "true",
"dynamic_partition.time_unit" = "DAY",
"dynamic_partition.start" = "-30",
"dynamic_partition.end" = "3",
"dynamic_partition.prefix" = "p"
); -- changelog.ttl_seconds is not set: one day
CREATE MATERIALIZED VIEW sales_by_day
BUILD IMMEDIATE REFRESH INCREMENTAL ON SCHEDULE EVERY 7 DAY STARTS
'2026-11-02 02:00:00'
DISTRIBUTED BY HASH(dt) BUCKETS 1
AS SELECT dt, COUNT(*) AS cnt, SUM(amount) AS total FROM orders GROUP BY dt;
```
Orders are loaded in batches during the day (09:00 to 18:00) into that day's
partition, and old partitions are not modified:
| Time | What happens |
| --- | --- |
| Mon 11-02 02:00, refresh | Last week's partitions are consumed and have
offsets. `p20261102` is still empty, so it has no offset |
| 11-02 to 11-08, daytime | Each day's orders are loaded into that day's
partition: `p20261102` to `p20261108` all have data, and none of them has an
offset |
| Mon 11-09 02:00, refresh | Only these seven partitions have new data, and
none of them has an offset to expire, so nothing is rejected. All seven are
read from the first TSO the changelog retains, which is 11-08 02:00 |
| Result | The orders of 11-02 to 11-07 are skipped and only 11-08 is read.
The refresh is SUCCESS, with an empty `IvmFallbackReason` |
After that refresh the seven partitions have offsets at their latest TSO, so
the six missing days are never read again; only a manual COMPLETE refresh
brings them back. The same happens every week. `SELECT dt, SUM(amount) FROM
orders GROUP BY dt` is rewritten to the MV and misses the same six days.
#### Example 2: refreshes interrupted for a few days
Same table, with the MV refreshed every night at 02:00. Normally every
partition is first read less than a day after its rows were written, so nothing
is lost. Now the refresh fails three nights in a row for an unrelated reason,
or the job is paused during an upgrade. When it succeeds on the fourth night,
the partitions of those days were never consumed: they have no offsets, they
are read from "now minus one day", and only the last day reaches the MV. What
the user sees is a refresh that recovered.
#### Example 3: a partition that existed, empty, when the MV was built; a
partitioned (PCT) MV is affected too
With `dynamic_partition.end = 3`, the partitions of the next three days
exist, empty, when the MV is created. Take an MV partitioned like its base
table:
```sql
CREATE MATERIALIZED VIEW sales_pmv
BUILD IMMEDIATE REFRESH INCREMENTAL ON SCHEDULE EVERY 7 DAY
PARTITION BY (dt)
DISTRIBUTED BY HASH(dt) BUCKETS 1
AS SELECT dt, order_id, amount FROM orders;
```
- The three pre-created partitions already have their MV partitions, so when
their data arrives they are caught up incrementally instead of rebuilt. Under
the timing of example 1 or 2 they lose rows.
- A partition created after the MV is a new base partition: the PCT MV
rebuilds its MV partition, so that one is not affected.
- A non-partitioned MV is affected by both kinds.
A LIST partition behaves the same way: `PARTITION p_eu VALUES IN ('EU')`
stays empty until the region launches, and if its first import lands more than
a TTL before the next refresh, that import never reaches the MV.
#### When it does not happen
- Refreshes always succeed on time, and the interval is shorter than the
TTL: every row is read within the TTL.
- The partition already has an offset. An expired offset stops the read of a
unique-key table: a failure today, and COMPLETE with #69772 when the refresh
may fall back. (A duplicate-key table clips an expired offset today; that is a
separate problem with its own fix.)
- In a PCT MV, a base partition added after the MV was built: its MV
partition is rebuilt.
- User streams: reading the changelog from the first retained TSO is their
semantics, and this PR keeps it.
The new regression test
[`test_ivm_partition_without_offset`](https://github.com/apache/doris/blob/7549bc2bb2184cee197e66ee0c905c3d50f272e9/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_without_offset.groovy)
compresses these timelines into a one-second TTL. On a build without this PR,
all three MVs in it end up without those rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
