yujun777 opened a new pull request, #69896:
URL: https://github.com/apache/doris/pull/69896
### What problem does this PR solve?
Issue Number: None
Related PR: #69850
Problem Summary:
**In short.** A table stream estimates the change rows a read returns from
the row count of the partition's changelog index: a read of the range between
the stream offset and the end of the read returns at most the change rows the
changelog still holds. The count the frontend holds comes from the tablet
reports, which a backend sends every `tablet_stat_update_interval_second` (60s
by default), so a read which follows a load within that interval sees the count
from before the load. The estimate then falls back to the row count of the
table: safe, but it answers with the whole table where the answer is a few
change rows.
The frontend cannot derive the change rows from the rows it loaded into the
base index: an update writes change rows without adding a base row, and a
delete adds none to the base but at least one to the changelog. The backend
knows them, so it now reports them.
**How this PR fixes it.**
- The publish task reports the delta rows of the changelog tablets as well,
not only of the base tablets: the report already visits the rowsets of every
tablet a transaction wrote, and a changelog tablet is one of them.
- `AnalysisManager.replayUpdateRowsRecord` routes those tablets by the index
each tablet belongs to (from the tablet inverted index, which the followers
have as well, so a replay reproduces the routing without a new journal entry),
and `TableStatsMeta` accumulates them per index (Gson field `"iur"`). The base
index rows are counted as before: the change rows are neither added to them nor
mistaken for them.
- `TableStatsMeta.getRowCountWithDeltaRows` returns the accumulated rows for
a changelog index: an analysis never collects its row count, so the rows of the
loads are the count there.
A stream read consumes this through
`TableStatsMeta.getRowCountWithDeltaRows`, which is where the estimate of
#69850 reads its counters from; until that lands, the recorded rows are visible
in the table statistics only.
### Release note
None
### Check List (For Author)
- Test: Unit test
- Unit test: `TableStatsMetaTest`, a new case (the rows a load writes to
the changelog index are accumulated per index and reported by
`getRowCountWithDeltaRows`, while the base index rows are not affected).
`TableStatsMetaTest` 16/16 and `AnalysisManagerTest` 27/27 pass with the change.
- No regression exercises the reporting window: it needs a load followed
by a read within the tablet reporting interval, which the suites cannot time
reliably. Every load now reports the change rows as well, so the existing
suites which read the row counts of a loaded table cover the base index side of
the change.
- Behavior changed: No user visible change; a load additionally reports the
rows it wrote to the changelog index.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
