englefly opened a new issue, #69280: URL: https://github.com/apache/doris/issues/69280
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master, e.g. `5a561bd6f98a`. The code path is present in every branch that contains the truncate handling in `AnalysisManager.replayUpdateRowsRecord()`. Found while reviewing #69028; it is not introduced by that PR. ### What's Wrong? `TableStatsMeta.updatedRows` is the "rows loaded since the row count was collected" delta that the planner adds to the collected row count when the backends have not reported the new tablets yet: - `TableStatsMeta.getBaseIndexDeltaRowCount()` = `updatedRows - baseline` - `StatsCalculator.getOlapTableRowCount()` falls back to `collected + delta` when `OlapTable.getRowCountForIndex(indexId, strict = true)` returns -1. Two partition-removal paths corrupt this counter: 1. **Partial `TRUNCATE PARTITION`**: `AnalysisManager.replayUpdateRowsRecord()` (master) adds the number of *removed* rows to `updatedRows`: ```java // Handle truncate table if (event.getPartitionToUpdateRows() != null && event.getTableId() > 0) { Map<Long, Long> partRows = event.getPartitionToUpdateRows(); long totalRows = partRows.values().stream().mapToLong(rows -> rows).sum(); TableStatsMeta statsStatus = idToTblStats.get(event.getTableId()); if (statsStatus != null) { statsStatus.updatedRows.addAndGet(totalRows); // removed rows are counted as loaded rows ``` The event is meant to signal "the table changed" to the auto-analyze, but the same live counter is later used as a *positive* row delta. 2. **`DROP PARTITION` reports nothing**: `InternalCatalog.dropPartitionWithoutCheck()` / `replayDropPartition()` only set `TableStatsMeta.partitionChanged = true` for auto-analyze. The dropped partition's rows stay in `updatedRows` (when they were loaded after the collection) and in the collected count (when they were collected before the drop). Both are pre-existing. #69028 makes the record exist for a never-analyzed table and gives it a baseline of 0, so the same counter now also becomes the cardinality of a never-analyzed table, which makes the problem much easier to hit; it does not create the counter corruption itself. ### What You Expected? Removing rows must not increase the row count estimate. The expected value is `collected - removed + (loaded - baseline)` (bounded below by 0). A partial truncate / drop should either adjust the load-derived delta or invalidate it (report "unknown" until the backends report the new tablets or an analyze recollects). ### How to Reproduce? ```sql CREATE TABLE t (k INT, v INT) DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p1 VALUES LESS THAN (100), PARTITION p2 VALUES LESS THAN (200) ) DISTRIBUTED BY HASH(k) BUCKETS 1; INSERT INTO t VALUES (1,1),(2,2); -- p1, 2 rows INSERT INTO t VALUES (101,1),(102,2); -- p2, 2 rows -- wait for tablet_stat_update_interval_second so that both partitions are reported TRUNCATE PARTITION p2; -- 2 rows are removed INSERT INTO t VALUES (103,3); -- the replacement partition, 1 row EXPLAIN SELECT * FROM t; -- actual: cardinality = 7 (record created by the load: collected = 0, updatedRows = 4 + 2 + 1) -- expected: cardinality = 3 (p1 has 2 rows + the 1 row loaded after the truncate) ``` The same happens when the table was analyzed before the truncate: `ANALYZE TABLE t` collects 4 rows (baseline = 4), `TRUNCATE PARTITION p2` makes `updatedRows = 6`, the following 1-row load makes it 7, and the fallback reports `4 + (7 - 4) = 7` instead of 3. For `DROP PARTITION`, dropping `p2` and loading one row keeps the 2 historical rows of `p2` in the delta as well. ### Anything Else? Root cause and possible fix direction (for discussion): - Keep the "changed rows" semantics used by auto-analyze, but do not reuse it as the row-count delta: track removed rows separately (e.g. `removedRows` plus a baseline captured together with `updatedRowsBase`) and use `(updatedRows - updatedRowsBase) - (removedRows - removedRowsBase)` as the delta, clamped so that the estimate never goes negative. - `DROP PARTITION` has no event at all; it needs to report the dropped partition's base-index row count (on master and on replay, like `TruncateTableInfo.updateRecords` does), or invalidate the record. - `TableStatsMeta.getRowCountWithDeltaRows()` (#69028) is the consumer for the row count; `StatsCalculator.computeDeltaRowCount()` uses the same value for the stale-statistics recovery of the filter estimation and should follow the same net semantics. - Cloud mode: the FE only receives `TableStatsPB{table_id, updated_row_count}` from the meta-service, so this fix is local-mode only unless a meta-service counterpart is added. - Tests: never-analyzed table (truncate + load), analyzed table (analyze -> truncate -> load), drop partition, multiple removals, and a removal followed by a backend report (the strict `getRowCountForIndex` path takes over). Related: #69028 (creates the table stats record when a load finishes), #69262 (temporary partitions are counted in the same delta). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
