englefly opened a new pull request, #68282:
URL: https://github.com/apache/doris/pull/68282
### What problem does this PR solve?
Issue Number: None (no upstream issue filed; the case is described below)
Problem Summary:
Reproduction (5 rows loaded into a 1 bucket duplicate table, then):
TRUNCATE TABLE t;
INSERT INTO t VALUES (1,1),(2,2),(3,3); -- immediately
EXPLAIN SELECT * FROM t; -- immediately
Before this change the scan of `t` is planned with `cardinality=1`, i.e.
`StatsCalculator.getOlapTableRowCount()` returns -1 even though the table is
not empty.
Root cause: `getOlapTableRowCount()` has two sources for the row count of a
table
1. `olapTable.getRowCountForIndex(selectedIndexId, true)`, which returns
-1 until the
backends have reported the row count of *every* tablet of the index.
After TRUNCATE
TABLE all tablets are new, and the row count loaded right after the
truncation is not
in that report either, so this returns -1 within a window of up to
`tablet_stat_update_interval_second` (60s by default);
2. `TableStatsMeta.getRowCount(indexId) +
TableStatsMeta.getBaseIndexDeltaRowCount()`,
which is the fallback used while (1) is -1.
`InternalCatalog.truncateTable()` removed the `TableStatsMeta` of the
truncated table. As a
consequence the fallback did not exist at all, and, more importantly, the
rows of every load
performed after the truncation were dropped by
`AnalysisManager.replayUpdateRowsRecord()`
(which only accumulates into an existing record), so the row count could not
be recovered
even later. The result was -1 with no way back.
Fix:
* `InternalCatalog.truncateTable()` calls
`AnalysisManager.resetTableStats()` instead of
`removeTableStats()` for a whole-table truncation. The record is reset to
the state of an
empty table (updatedRows = 0, indexesRowCount = 0 for every index,
partitionUpdateRows and
the per column stats cleared, partitionChanged and userInjected reset) and
kept, so that the
rows loaded afterwards are accumulated and reported as the delta row count
of the table. A
record is created when the table does not have one yet. The reset is
journaled, so the
followers and the image stay consistent with the master.
* `TableStatsMeta.getBaseIndexDeltaRowCount()` no longer returns 0 for an
empty
`colToColStatsMeta`. An empty map means no baseline has ever been
collected, which is
exactly the state of a reset record, so all the rows of `updatedRows` are
the delta. Without
this, the delta of a reset record would have been `updatedRows - <row
count captured by the
analysis before the truncation>`, a negative number. The `userInjected`
guard is kept, which
is the only way an empty map was reached before (a record created by
`ALTER TABLE ... SET STATS` has empty job columns).
* `SHOW TABLE STATS` now prints an empty timestamp instead of `1970-01-01
08:00:00` for a
record which was never analyzed, which is now possible for a table
truncated before being
analyzed.
TRUNCATE TABLE still drops the collected statistics themselves (the column
stats rows and the
in-memory cache), only the record is kept; the `columns` column of `SHOW
TABLE STATS` is
therefore `[]` instead of empty, and the existing
`test_drop_stats_and_truncate` case is
updated accordingly.
End to end result of the reproduction above (manual test on a real cluster,
single BE,
`EXPLAIN` prints the cardinality of the scan):
before: cardinality=1 (getOlapTableRowCount() == -1)
after: cardinality=3
The same holds when the table was never analyzed before the truncation
(`TRUNCATE TABLE` + load 4 rows -> `cardinality=4`).
### Release note
The row count of a table is no longer reported as unknown (-1) after
`TRUNCATE TABLE`
followed by an immediate load and query.
### Check List (For Author)
- Test: Regression test
(`regression-test/suites/statistics/test_row_count_after_truncate.groovy`,
run against a local cluster: passed), updated
`test_drop_stats_and_truncate.groovy`
(passed), Unit test (`TableStatsMetaTest`, 4 tests, passed), Manual test
on a local
FE + BE cluster for the before/after numbers above.
- Behavior changed: Yes. `TRUNCATE TABLE` keeps the table stats record
instead of deleting
it, so `SHOW TABLE STATS` shows a record with `columns=[]` after a
truncation, and the
row count of the rows loaded after the truncation is reported to the
optimizer instead of
-1. Column statistics are still dropped as before.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
